Verified (Simple) Backups

Yeah, yeah… “vsb”, not “veb”. But I’m a sucker for psuedo-word commands.

enough for two 82-width buffers (set-frame-size (selected-frame) 170 69)

http://www.gnu.org/software/emacs/manual/html_node/org/Clocking-work-time.html

Go links

A Tour of Go
- http://tour.golang.org/#1
- http://127.0.0.1:3999 (gotta run gotour from cmd line first)
Effective Go
- http://golang.org/doc/effective_go.html
- lots of useful info about most everything there
packages
- http://golang.org/pkg/
- docs & source
Gob
- http://blog.golang.org/2011/03/gobs-of-data.html
- http://golang.org/pkg/encoding/gob/#Register

Mysteries of Go

Don’t have two mains inside a package.

A main of veb.go in package veb works. A main of cmd.go in package veb gives BS mystery errors. palladium:veb spydez$ go run src/veb/cmd.go

src/veb/checksum.go:16: undefined: IndexEntry

there is nothing wrong with checksum: mv src/veb/cmd.go src/veb/veb.go go run src/veb/veb.go ((normal, expected output from running main))

But this is for ‘go run’. go install will whine when it finds main & package together regardless.

“xsum.go” is a no-no? It worked as hash.go, didn’t as xsum.go, and does as checksum.go.

select + closed channel == WTF

it’ll run the closed channel’s case randomly. 0.o

CMDs

some stuff in test/scripts

Check that veb is finding all files:

obviously veb needs to print out only file names

(go run src/veb/veb.go && find test/scratch -type f | sed “s,test/scratch/,,g”) | sort | uniq -c

making tree of files from text file w/ filenames in it… find . -type f -iname > path/to/walkfiles.txt cat ../../../walkfiles.txt | perl -p -e “s/\n/\0/;” > ../../../walknull.txt cat ../../../walkdirnull.txt | xargs -0 mkdir -p cat ../../../walknull.txt | xargs -0 -I{} sh -c ‘echo $1 > $1.txt’ – {} TODO: Make these 2-3 MB files of random data…

randomed files… http://linuxcommando.blogspot.com/2008/06/create-file-of-given-size-with-random.html

but OSX doesn’t like “1M”… dd if=/dev/urandom of=test/data/hash/rand2mb.bin bs=1048576 count=2

few random files w/ checksum file for f in `seq 1 15`; do dd if=/dev/urandom of=rand_${f}.bin bs=1048576 count=2; done for f in `seq 1 15`; do shasum -a1 rand_${f}.bin ; done > ../../verification/short-walk-files.txt shasum -a1 –check ../../verification/short-walk-files.txt

users.txt

Costs

TODOs

line numbers (linum) in Go-mode buffers

Is fill-column not default 80 in my dotfiles?!

closed: [2012-05-15 Tue 19:16]

Looks like it is… shrugs

install HG for go-get

closed: [2012-05-08 Tue 15:47]

install from MacPorts

update Xcode for MacPorts for HG

closed: [2012-05-08 Tue 15:28]

TASKS

000: Go setup

closed: [2012-05-08 Tue 22:43]

GOPATH, dir structure, etc.
- ignore pkg & bin dirs in git

001: walk dir tree, print out

closed: [2012-05-08 Tue 22:48]

make test tree w/ tons o’ files.
- print out some of files in iTunes lib, use to make mostly empty files of same layout.
  - make sure it uses the crazy letters (eg Folie À Deux), not just ACII.

002: SHA-256 vs SHA1

closed: [2012-05-08 Tue 23:50]

shasum for initial stab…

SHA 1:

893M file (935905255 bytes)
0m52.194s real time as per `time shasum -pa 1 SCOTT_PILGRIM.m4v`
17.1006017 MBps

SHA 1 (take 2): palladium:Movies spydez$ ls -lh unwatched/MOVIE_2.iso -rw-r–r–@ 1 spydez staff 7.9G May 2 15:23 unwatched/MOVIE_2.iso palladium:Movies spydez$ ls -l unwatched/MOVIE_2.iso -rw-r–r–@ 1 spydez staff 8465022976 May 2 15:23 unwatched/MOVIE_2.iso palladium:Movies spydez$ time shasum -pa 1 unwatched/MOVIE_2.iso e453b6e10aff36ce5dca0506366ac9b4ee1e3f3e ?unwatched/MOVIE_2.iso

real 4m14.084s (+ (* 4 60) 14.084) 254.084 sec user 0m34.527s sys 0m16.014s

SHA 256: palladium:Movies spydez$ ls -lh unwatched/MOVIE_1.iso -rw-r–r–@ 1 spydez staff 7.9G May 2 15:15 unwatched/MOVIE_1.iso palladium:Movies spydez$ ls -l unwatched/MOVIE_1.iso -rw-r–r–@ 1 spydez staff 8533966848 May 2 15:15 unwatched/MOVIE_1.iso palladium:Movies spydez$ time shasum -pa 256 unwatched/MOVIE_1.iso b2a525dde826e301177b78479d26367d2da8252a36eb83109d7477f62fa2416f ?unwatched/MOVIE_1.iso

real 10m53.673s (+ (* 10 60) 53.673) 653.673 sec user 1m18.879s sys 0m14.157s

SHA 1 : 31.772465 MBps SHA 256: 12.450606 MBps

And now… the easier way. http://www.cryptopp.com/benchmarks.html SHA-1 is faster.

53 sec vs 82 sec for 8.9 GB file

CONCLUSION: Use SHA-1, since speed is more important than security

But possibly make hashes user choosable in case someone hates SHA-1

003: Hash file, print out result/time/etc

closed: [2012-05-09 Wed 00:32]

That was pretty easy…

004: walk & hash

closed: [2012-05-09 Wed 16:09]

Made some rand files & checksummed them. (CMDs -> few random)

Steps:

[X] Make walk & hash non-main
[X] Make lil’ veb its very own main.

005: time stuff

closed: [2012-05-18 Fri 19:57]

timer.go done!

006: goroutines

closed: [2012-05-18 Fri 19:41]

launcher script to set Go with the right number of procs?
time walk w/ goroutines vs single-thread

007: meta-file format

closed: [2012-05-09 Wed 17:29]

See git’s. They know what they’re doing.
- http://git-scm.com/book/en/Git-Internals
how git determines if a file’s modified:
- http://www-cs-students.stanford.edu/~blynn/gitmagic/ch08.html#_indexing

possibly use shasum’s checkfile method for the xsums instead of git tree object’s

and then something like the git index for checking modifications

So 2 meta-files.

.veb/index – file info for quickly finding new/deleted/modified files
.veb/xsums – checksums of all known files.

008: index file

Go os.FileInfo gives: Name() string // base name of the file Size() int64 // length in bytes for regular files; system-dependent for others Mode() FileMode // file mode bits ModTime() time.Time // modification time http://golang.org/pkg/os/#FileInfo

So use just those for now. Git index has a ton more, but this is simpler and the above should do. http://git.rsbx.net/Documents/Git_Data_Formats.txt

Will need full (relative) path instead of just basename to avoid same-named files …like my “project.org” files.

Since this is mostly to learn Go, will be using the gob pkg to write file info out/read back in.

http://golang.org/pkg/encoding/gob/

TODOTODOTODO

TODO [4/5]
- [X] Make add() function for adding new files to Index
  - make checkWalker() use it?
    - No… add new things to Index after they’re backed up.
- [X] remove addWalker()
- [X] remove build()
- [ ] test index.go!
- [X] implement Export()? Or is that for 009:xsums?
  - Xsums.

009: xsums file

closed: [2012-05-20 Sun 21:55]

use shasum’s checkfile format for the xsums

or add to index?
- or both…

TODOTODOTODO

TODO [0/2]
- [ ] implement Export() for sending xsums out to shasum-formatted file.
  - …I’m thinking this should go back in Index. :/
    - maybe have the formatter in Xsum.
    - Put it in index.Save()

010: log [3/5]

Use Go log pkg to log stuff.

Is there a way to tee the logs to stderr and a log file?
- I think I saw a MultiWriter somewhere…
  - http://golang.org/pkg/io/#MultiWriter

[X] index.go
[X] checksum.go
[X] veb.go (commands)
[ ] MultiWriter

Log levels?

struct w/ 3 logs
- error
- warning
- info

Yes.

011: parallize

closed: [2012-05-20 Sun 21:58]

http://golang.org/doc/effective_go.html#parallel

put that code in veb.go before stuff happens.

straight walk vs goroutined walk?

filepath.Walk itself is serial, so no big benefits can be gained

serial walk: 32 ms for 2752 files (3076 files & folders)

think we’ll be fine for now.

012: bash scripts for testing [4/8]

[X] make scripts create in scratch/local or scratch/remote instead of just scratch
[X] script to clean up test area
- something simple should do, like: rm -rf test/scratch
[X] script to make small tree
[X] script to modify something in small tree
[ ] script to just bump mtime (touch)?
[ ] script to make full tree
[ ] script to modify things in small tree
[ ] script to just bump mtime (touch)?

013: !!! use channels of IndexEntries

closed: [2012-05-19 Sat 02:09]

so everything can be sent together, and won’t have to restat files or anything

014: commands [2/2]

[X] test ‘veb status’ with changed files
- and changed & new files
[X] test ‘veb verify’ on freshly init’d repo

init

closed: [2012-05-19 Sat 15:40]

make .veb dir and files, doesn’t really do anything
git init

status

closed: [2012-05-19 Sat 19:20]

checks what’s changed/new, doesn’t do anything
git status

GIT STATUS OUTPUT: palladium:scratch spydez$ git status

#

DESIRED VEB STATUS OUTPUT: palladium:scratch spydez$ veb status veb repository at /path/to/here

Changed files:

foo/bar/baz.bin

filesize increased 400 bytes (90.3MB -> 90.3MB)
modified on (2012-05-19 16:11:05)

foo/quux.mp3

modification time only (2012-05-19 16:11:05)

New files:

xyzzy.iso

8.9GB, modified on (2012-05-19 16:11:05)

firefly.m4v

80MB, modified on (2012-05-19 16:11:05)

MAKE SURE CHANGED FILES ARE THINGS YOU’VE ACTUALLY CHANGED (use “veb fix <file>” if a file has been corrupted in this repository) (use “veb push”, “veb pull”, or “veb sync” to commit changed/new files)

actual output (all new; no changed files) palladium:scratch spydez$ go run ../../src/veb/veb.go status info >> 2012/05/19 18:59:08 veb.go:155: START status veb repository at /Users/spydez/sourcepan/veb/test/scratch

New files:

.DS_Store

12.00KB, modified on (2012-05-18 17:47:19 -0500 CDT)

311/.DS_Store

6.00KB, modified on (2012-05-18 17:47:14 -0500 CDT)

311/Don’t Tread On Me/01 Don’t Tread On Me.test.bin

2.00MB, modified on (2012-05-18 17:47:04 -0500 CDT)

311/Don’t Tread On Me/02 Thank Your Lucky Stars.test.bin

2.00MB, modified on (2012-05-18 17:47:04 -0500 CDT)

311/Don’t Tread On Me/09 Whiskey & Wine.test.bin

2.00MB, modified on (2012-05-18 17:47:04 -0500 CDT)

311/Don’t Tread On Me/10 It’s Getting OK Now.test.bin

2.00MB, modified on (2012-05-18 17:47:04 -0500 CDT)

311/Don’t Tread On Me/11 There’s Always An Excuse.test.bin

2.00MB, modified on (2012-05-18 17:47:04 -0500 CDT)

Ace Troubleshooter/.DS_Store

6.00KB, modified on (2012-05-18 17:47:14 -0500 CDT)

Ace Troubleshooter/It’s Never Enough/01 Ball & Chain.test.bin

2.00MB, modified on (2012-05-18 17:47:04 -0500 CDT)

Ace Troubleshooter/It’s Never Enough/02 Anything.test.bin

2.00MB, modified on (2012-05-18 17:47:05 -0500 CDT)

Apt° Core/.DS_Store

6.00KB, modified on (2012-05-18 17:47:14 -0500 CDT)

Apt° Core/2/01 No Such Thing As Time.test.bin

2.00MB, modified on (2012-05-18 17:47:05 -0500 CDT)

Fall Out Boy/.DS_Store

6.00KB, modified on (2012-05-18 17:47:09 -0500 CDT)

Fall Out Boy/Folie À Deux/01 Disloyal Order Of Water Buffaloes.test.bin

2.00MB, modified on (2012-05-18 17:53:11 -0500 CDT)

Fall Out Boy/Folie À Deux/02 I Don’t Care.test.bin

2.00MB, modified on (2012-05-18 17:53:11 -0500 CDT)

Fall Out Boy/Folie À Deux/03 She’s My Winona.test.bin

2.00MB, modified on (2012-05-18 17:47:05 -0500 CDT)

Fall Out Boy/Folie À Deux/04 America’s Suitehearts.test.bin

2.00MB, modified on (2012-05-18 17:47:05 -0500 CDT)

Fall Out Boy/Folie À Deux/05 Headfirst Slide Into Cooperstown On A Bad Bet.test.bin

2.00MB, modified on (2012-05-18 17:47:06 -0500 CDT)

Five Iron Frenzy/.DS_Store

6.00KB, modified on (2012-05-18 17:47:14 -0500 CDT)

Five Iron Frenzy/Our Newest Album Ever!/01 Handbook for the Sellout.test.bin

2.00MB, modified on (2012-05-18 17:53:11 -0500 CDT)

Five Iron Frenzy/Our Newest Album Ever!/02 Where is Micah_.test.bin

2.00MB, modified on (2012-05-18 17:47:06 -0500 CDT)

Five Iron Frenzy/Our Newest Album Ever!/11 Oh, Canada.test.bin

2.00MB, modified on (2012-05-18 17:47:06 -0500 CDT)

Five Iron Frenzy/Our Newest Album Ever!/12 Most Likely to Succeed.test.bin

2.00MB, modified on (2012-05-18 17:47:06 -0500 CDT)

Five Iron Frenzy/Our Newest Album Ever!/13 Every New Day.test.bin

2.00MB, modified on (2012-05-18 17:47:06 -0500 CDT)

Movits!/.DS_Store

6.00KB, modified on (2012-05-18 17:47:09 -0500 CDT)

Movits!/Äppelknyckarjazz/01 Ta på dig dansskorna.test.bin

2.00MB, modified on (2012-05-18 17:53:11 -0500 CDT)

Movits!/Äppelknyckarjazz/03 Swing för hyresgästföreningen.test.bin

2.00MB, modified on (2012-05-18 17:47:07 -0500 CDT)

Movits!/Äppelknyckarjazz/04 Fast tvärtom.test.bin

2.00MB, modified on (2012-05-18 17:47:07 -0500 CDT)

Movits!/Äppelknyckarjazz/06 Fel del av gården.test.bin

2.00MB, modified on (2012-05-18 17:47:07 -0500 CDT)

Movits!/Äppelknyckarjazz/08 Tom Jones.test.bin

2.00MB, modified on (2012-05-18 17:47:07 -0500 CDT)

Movits!/Äppelknyckarjazz/09 Äppelknyckarjazz.test.bin

2.00MB, modified on (2012-05-18 17:47:07 -0500 CDT)

Movits!/Äppelknyckarjazz/10 Stick iväg Jack del II.test.bin

2.00MB, modified on (2012-05-18 17:47:07 -0500 CDT)

Movits!/Äppelknyckarjazz/11 2 dollar på fickan.test.bin

2.00MB, modified on (2012-05-18 17:47:08 -0500 CDT)

Parov Stelar/.DS_Store

6.00KB, modified on (2012-05-18 17:47:19 -0500 CDT)

Parov Stelar/Coco Pt.1/01 Coco (Featuring Lilja Bloom).test.bin

2.00MB, modified on (2012-05-18 17:47:08 -0500 CDT)

Parov Stelar/Coco Pt.1/02 Hurt.test.bin

2.00MB, modified on (2012-05-18 17:47:08 -0500 CDT)

Parov Stelar/Coco Pt.1/03 For Rose(수원 아이파크 시티 CF삽입곡).test.bin

2.00MB, modified on (2012-05-18 17:47:08 -0500 CDT)

Parov Stelar/Coco Pt.2/01 The Mojo Radio Gang(Radio Ver.).test.bin

2.00MB, modified on (2012-05-18 17:47:08 -0500 CDT)

Parov Stelar/Coco Pt.2/02 Ragtime Cat(Featuring Lilja Bloom).test.bin

2.00MB, modified on (2012-05-18 17:47:08 -0500 CDT)

Parov Stelar/Coco Pt.2/03 Silent Snow(Featuring Max The Sax).test.bin

2.00MB, modified on (2012-05-18 17:47:08 -0500 CDT)

Parov Stelar/Coco Pt.2/04 Libella Swing(현대카드 TV CF 삽입곡).test.bin

2.00MB, modified on (2012-05-18 17:53:11 -0500 CDT)

Trans-Siberian Orchestra/.DS_Store

6.00KB, modified on (2012-05-18 17:47:19 -0500 CDT)

Trans-Siberian Orchestra/Christmas Eve and Other Stories/02 O Come All Ye Faithful_O Holy Night (instrumental).test.bin

2.00MB, modified on (2012-05-18 17:47:09 -0500 CDT)

Trans-Siberian Orchestra/Christmas Eve and Other Stories/05 The Silent Nutcracker (Instrumental).test.bin

2.00MB, modified on (2012-05-18 17:47:09 -0500 CDT)

Trans-Siberian Orchestra/Christmas Eve and Other Stories/06 A Mad Russian’s Christmas (Instrumental).test.bin

2.00MB, modified on (2012-05-18 17:47:09 -0500 CDT)

MAKE SURE CHANGED FILES ARE THINGS YOU’VE ACTUALLY CHANGED (use “veb fix <file>” if a file has been corrupted in this repository) (use “veb push”, “veb pull”, or “veb sync” to commit changed/new files)

summary: 45 new, 0 changed info >> 2012/05/19 18:59:08 veb.go:331: END status info >> 2012/05/19 18:59:08 veb.go:333: status (45 new, 0 changed) took 3.161ms

actual output (nothing to do) palladium:local spydez$ go run ../../../src/veb/veb.go status info >> 2012/05/20 17:03:34 veb.go:252: START status veb repository at /Users/spydez/sourcepan/veb/test/scratch/local

No changes or new files.

summary: 0 new, 0 changed info >> 2012/05/20 17:03:34 veb.go:387: END status info >> 2012/05/20 17:03:34 veb.go:389: status (0 new, 0 changed) took 1.802ms

actual output (no new; some changed files) palladium:local spydez$ go run ../../../src/veb/veb.go status info >> 2012/05/20 17:17:55 veb.go:252: START status veb repository at /Users/spydez/sourcepan/veb/test/scratch/local

Changed files:

311/Don’t Tread On Me/01 Don’t Tread On Me.test.bin

file mode changed (644-> 664)

Fall Out Boy/Folie À Deux/01 Disloyal Order Of Water Buffaloes.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)
file mode changed (644-> 664)

Fall Out Boy/Folie À Deux/02 I Don’t Care.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)

Five Iron Frenzy/Our Newest Album Ever!/01 Handbook for the Sellout.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)

Movits!/Äppelknyckarjazz/01 Ta på dig dansskorna.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)

Parov Stelar/Coco Pt.2/04 Libella Swing(현대카드 TV CF 삽입곡).test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)

MAKE SURE CHANGED FILES ARE THINGS YOU’VE ACTUALLY CHANGED (use ‘veb fix <file>’ if a file has been corrupted in this repository) (use ‘veb push’, ‘veb pull’, or ‘veb sync’ to commit changed/new files)

summary: 0 new, 6 changed info >> 2012/05/20 17:17:55 veb.go:387: END status info >> 2012/05/20 17:17:55 veb.go:389: status (0 new, 6 changed) took 2.203ms

actual output (new & changed) palladium:local spydez$ go run ../../../src/veb/veb.go status info >> 2012/05/20 17:47:38 veb.go:254: START status veb repository at /Users/spydez/sourcepan/veb/test/scratch/local

New files:

foo.new

0.00B, modified on (2012-05-20 17:47:34 -0500 CDT)

Changed files:

311/Don’t Tread On Me/01 Don’t Tread On Me.test.bin

file mode changed (-rw-r–r– -> -rw-rw-r–)

Fall Out Boy/Folie À Deux/01 Disloyal Order Of Water Buffaloes.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)
file mode changed (-rw-r–r– -> -rw-rw-r–)

Fall Out Boy/Folie À Deux/02 I Don’t Care.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)

Five Iron Frenzy/Our Newest Album Ever!/01 Handbook for the Sellout.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)

Movits!/Äppelknyckarjazz/01 Ta på dig dansskorna.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)

Parov Stelar/Coco Pt.2/04 Libella Swing(현대카드 TV CF 삽입곡).test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)

MAKE SURE CHANGED FILES ARE THINGS YOU’VE ACTUALLY CHANGED (use ‘veb fix <file>’ if a file has been corrupted in this repository) (use ‘veb push’, ‘veb pull’, or ‘veb sync’ to commit changed/new files)

summary: 1 new, 6 changed info >> 2012/05/20 17:47:38 veb.go:386: END status info >> 2012/05/20 17:47:38 veb.go:388: status (1 new, 6 changed) took 37.408ms

verify [2/2]

closed: [2012-05-20 Sun 18:26]

TODO

[X] stats for status line & exit info log
[X] need ‘veb push’ or some other means of saving index.
- verifying nothing (empty index) is pretty bland.
- Made test-commit.
runs xsum on all files, warns when stuff’s different but not modified.
need a “quit” command for early exiting.

Firstly, do the quit early part.

DESIRED VEB STATUS OUTPUT: palladium:scratch spydez$ veb verify veb repository at /path/to/here

Verifying file checksums against those stored in veb index… Note: new files (as shown by ‘veb status’) will not be checked.

Changed files:

foo/bar/baz.bin

filesize increased 400 bytes (90.3MB -> 90.3MB)
modified on (2012-05-19 16:11:05)
previous SHA1: 79bac1510de7be376f89715635f7be51b006c59a
current SHA1: e58c8e3e4cd756d8b7716b022ece9181b1fd52da

scanned: 400 of 56780 files (1 changed file) (type ‘q’ to quit): _

actual output (full check) palladium:local spydez$ go run ../../../src/veb/veb.go verify info >> 2012/05/20 18:27:43 veb.go:396: START verify veb repository at /Users/spydez/sourcepan/veb/test/scratch/local

Verifying file checksums against those stored in veb index… Note: new files (as shown by ‘veb status’) will not be checked.

Files with new hashes:

Fall Out Boy/Folie À Deux/02 I Don’t Care.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)
previous SHA1: 1b206714df8b9c019df4ccd4aadafcae299d57a2
current SHA1: 7eadac6d4623b095daa8eeda44fe8a503bdb40d6

Fall Out Boy/Folie À Deux/01 Disloyal Order Of Water Buffaloes.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)
file mode changed (-rw-r–r– -> -rw-rw-r–)
previous SHA1: 86ac6e4f783113c8afd7dd8e32b50407885e9ea9
current SHA1: fa3ee41f9141f3f137b2846b50f3b46f69fd91a8

Movits!/Äppelknyckarjazz/01 Ta på dig dansskorna.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)
previous SHA1: 2aad0943ffa8306d030ea5d750ca9b780f69ec4c
current SHA1: 55e874427b6829801dd1a1e8a8532f9110a9c396

Parov Stelar/Coco Pt.2/04 Libella Swing(현대카드 TV CF 삽입곡).test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)
previous SHA1: f42f01fed7e4306fbcb0d3f954f1b0a2a23e82cf
current SHA1: aaa72c136960e5671bc6bdc9fcc42add05dea7f3

Five Iron Frenzy/Our Newest Album Ever!/01 Handbook for the Sellout.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)
previous SHA1: 418201374e41fc7e6be83e1e4c97acc24dd37cbc
current SHA1: 1752eaab329a3e9de4b31e0dd443af14569918d3

scanned: 36 of 36 files (5 changed) (type ‘q’ to quit):

MAKE SURE CHANGED FILES ARE THINGS YOU’VE ACTUALLY CHANGED (use ‘veb fix <file>’ if a file has been corrupted in this repository) (use ‘veb push’, ‘veb pull’, or ‘veb sync’ to commit changed/new files)

summary: 31 ok, 5 changed, 0 not checked info >> 2012/05/20 18:27:46 veb.go:473: END verify info >> 2012/05/20 18:27:46 veb.go:475: verify (31 ok, 5 changed, 0 not checked) took 3.122954s

actual ouput (early quit) palladium:local spydez$ go run ../../../src/veb/veb.go verify info >> 2012/05/20 18:28:13 veb.go:396: START verify veb repository at /Users/spydez/sourcepan/veb/test/scratch/local

Verifying file checksums against those stored in veb index… Note: new files (as shown by ‘veb status’) will not be checked.

Files with new hashes:

Movits!/Äppelknyckarjazz/01 Ta på dig dansskorna.test.bin

filesize increased 4.00B (2.00MB -> 2.00MB)
modified on (2012-05-20 17:13:48 -0500 CDT)
previous SHA1: 2aad0943ffa8306d030ea5d750ca9b780f69ec4c
current SHA1: 55e874427b6829801dd1a1e8a8532f9110a9c396

scanned: 12 of 36 files (1 changed) (type ‘q’ to quit): q scanned: 12 of 36 files (1 changed) (type ‘q’ to quit):

MAKE SURE CHANGED FILES ARE THINGS YOU’VE ACTUALLY CHANGED (use ‘veb fix <file>’ if a file has been corrupted in this repository) (use ‘veb push’, ‘veb pull’, or ‘veb sync’ to commit changed/new files)

summary: 11 ok, 1 changed, 24 not checked info >> 2012/05/20 18:28:14 veb.go:473: END verify info >> 2012/05/20 18:28:14 veb.go:475: verify (11 ok, 1 changed, 24 not checked) took 863.801ms

commit

closed: [2012-05-20 Sun 17:03]

debug command. For testin’.

Probably not ‘veb commit’. Maybe ‘veb test-commit’.

remote

closed: [2012-05-20 Sun 23:21]

changes backup location
silimalish to git remote

push

closed: [2012-05-21 Mon 22:00]

veb push - send local changes to remote

help

print help info

fix

for getting a good version of a corrupted file from the remote

veb fix path/to/file.bin

commit again

Turns out I probably want a commit. For when the backup exists already. local-dir$ veb init local-dir$ cd remotedir remotedir$ veb init remotedir$ veb commit remotedir$ cd local-dir local-dir$ veb remote remotedir local-dir$ veb push

Or maybe have push not just check for new stuff? Maybe force commit, then push ala git’s flow.

pull/sync

copies files to backup location

Probably don’t use “commit”. This is more like git’s push/pull.

veb pull - get remote changes to here
veb sync - veb push && veb pull w/ warnings about collisions
- maybe they should all warn

015: put globals somewhere global

closed: [2012-05-21 Mon 22:01]

metadata dir, index file name, xsum file name log currently in index.go and cmd.go

016: clean deleted files out of index

017: FINALS [2/5]

[ ] clean up
[X] write README
[X] go install, test the bin.
[ ] git tag v0.1

999: update commands: remote

make veb remote more like git remote

probably also pull remote out of index and put in config file.

$ git remote $ git remote add github git@github.com:spydez/hw.git $ git remote github $ git remote -v github git@github.com:spydez/hw.git (fetch) github git@github.com:spydez/hw.git (push)

999: update commands: push

make veb push more like git push

git push <remote-name>

999: nice

Can we make go run nice’d, or low priority?

999: future features

move commands out of main package so veb can be integrated into other things?
- Doubt it ever will… but might as well.
parallelized walk
- make walk only current dir, spawn subdirs off as separate goroutines
better multi-point syncing?
- eg:
  - desktop changes x.mp3
  - desktop backs up to NAS
  - laptop changes same x.mp3 (in a different manner… different song metadata or something)
  - laptop backs up to NAS
    - veb notifies user that a modified x.mp3 exists both on laptop and NAS and asks which is desired.
- may involve version controlling the veb meta-files and using git to figure that out.
  - “Hm… x.mp3 changed. Git! Do source and dest have previous file hases in common?”
choice of hash functions on init
- SHA1. SHA256. MD5 for the speed freaks…
non-local/non-mounted remotes
- talk w/ gob over network for index
- rsync files

example

palladium:scratch spydez$ cd local

palladium:local spydez$ veb init Initialized empty veb repository at ~/test/scratch/local

palladium:local spydez$ veb commit veb repository at ~/test/scratch/local

Committed files:

.DS_Store 311/Don’t Tread On Me/09 Whiskey & Wine.test.bin 311/Don’t Tread On Me/10 It’s Getting OK Now.test.bin 311/Don’t Tread On Me/02 Thank Your Lucky Stars.test.bin 311/Don’t Tread On Me/11 There’s Always An Excuse.test.bin Fall Out Boy/Folie À Deux/03 She’s My Winona.test.bin Ace Troubleshooter/It’s Never Enough/02 Anything.test.bin Fall Out Boy/Folie À Deux/01 Disloyal Order Of Water Buffaloes.test.bin 311/Don’t Tread On Me/01 Don’t Tread On Me.test.bin Apt° Core/2/01 No Such Thing As Time.test.bin Ace Troubleshooter/It’s Never Enough/01 Ball & Chain.test.bin Fall Out Boy/Folie À Deux/02 I Don’t Care.test.bin Fall Out Boy/Folie À Deux/04 America’s Suitehearts.test.bin Five Iron Frenzy/Our Newest Album Ever!/01 Handbook for the Sellout.test.bin Fall Out Boy/Folie À Deux/05 Headfirst Slide Into Cooperstown On A Bad Bet.test.bin Five Iron Frenzy/Our Newest Album Ever!/11 Oh, Canada.test.bin Movits!/Äppelknyckarjazz/01 Ta på dig dansskorna.test.bin Five Iron Frenzy/Our Newest Album Ever!/02 Where is Micah_.test.bin Five Iron Frenzy/Our Newest Album Ever!/13 Every New Day.test.bin Five Iron Frenzy/Our Newest Album Ever!/12 Most Likely to Succeed.test.bin Movits!/Äppelknyckarjazz/06 Fel del av gården.test.bin Movits!/Äppelknyckarjazz/03 Swing för hyresgästföreningen.test.bin Movits!/Äppelknyckarjazz/04 Fast tvärtom.test.bin Movits!/Äppelknyckarjazz/08 Tom Jones.test.bin Movits!/Äppelknyckarjazz/09 Äppelknyckarjazz.test.bin Movits!/Äppelknyckarjazz/11 2 dollar på fickan.test.bin bar.txt foo.txt Movits!/Äppelknyckarjazz/10 Stick iväg Jack del II.test.bin Parov Stelar/Coco Pt.1/01 Coco (Featuring Lilja Bloom).test.bin Parov Stelar/Coco Pt.2/03 Silent Snow(Featuring Max The Sax).test.bin Parov Stelar/Coco Pt.2/02 Ragtime Cat(Featuring Lilja Bloom).test.bin Parov Stelar/Coco Pt.2/01 The Mojo Radio Gang(Radio Ver.).test.bin Parov Stelar/Coco Pt.1/02 Hurt.test.bin Trans-Siberian Orchestra/Christmas Eve and Other Stories/05 The Silent Nutcracker (Instrumental).test.bin Parov Stelar/Coco Pt.1/03 For Rose(수원 아이파크 시티 CF삽입곡).test.bin Parov Stelar/Coco Pt.2/04 Libella Swing(현대카드 TV CF 삽입곡).test.bin Trans-Siberian Orchestra/Christmas Eve and Other Stories/06 A Mad Russian’s Christmas (Instrumental).test.bin Trans-Siberian Orchestra/Christmas Eve and Other Stories/02 O Come All Ye Faithful_O Holy Night (instrumental).test.bin

summary: 39 commits, 0 errors in 1.553082s

palladium:local spydez$ mkdir ../remote palladium:local spydez$ cd ../remote

palladium:remote spydez$ veb init Initialized empty veb repository at ~/test/scratch/remote

palladium:remote spydez$ cd ../local

palladium:local spydez$ veb remote ../remote veb repository at ~/test/scratch/local

veb added ~/test/scratch/remote as the remote

palladium:local spydez$ veb push veb repository at ~/test/scratch/local

Pushed files:

311/Don’t Tread On Me/10 It’s Getting OK Now.test.bin Movits!/Äppelknyckarjazz/09 Äppelknyckarjazz.test.bin Fall Out Boy/Folie À Deux/02 I Don’t Care.test.bin foo.txt Parov Stelar/Coco Pt.1/03 For Rose(수원 아이파크 시티 CF삽입곡).test.bin Ace Troubleshooter/It’s Never Enough/02 Anything.test.bin Fall Out Boy/Folie À Deux/04 America’s Suitehearts.test.bin Parov Stelar/Coco Pt.2/03 Silent Snow(Featuring Max The Sax).test.bin Fall Out Boy/Folie À Deux/05 Headfirst Slide Into Cooperstown On A Bad Bet.test.bin bar.txt Movits!/Äppelknyckarjazz/06 Fel del av gården.test.bin Trans-Siberian Orchestra/Christmas Eve and Other Stories/02 O Come All Ye Faithful_O Holy Night (instrumental).test.bin Fall Out Boy/Folie À Deux/01 Disloyal Order Of Water Buffaloes.test.bin Fall Out Boy/Folie À Deux/03 She’s My Winona.test.bin Parov Stelar/Coco Pt.1/02 Hurt.test.bin Five Iron Frenzy/Our Newest Album Ever!/02 Where is Micah_.test.bin .DS_Store Ace Troubleshooter/It’s Never Enough/01 Ball & Chain.test.bin Five Iron Frenzy/Our Newest Album Ever!/13 Every New Day.test.bin Parov Stelar/Coco Pt.2/04 Libella Swing(현대카드 TV CF 삽입곡).test.bin Movits!/Äppelknyckarjazz/03 Swing för hyresgästföreningen.test.bin Five Iron Frenzy/Our Newest Album Ever!/12 Most Likely to Succeed.test.bin Movits!/Äppelknyckarjazz/10 Stick iväg Jack del II.test.bin Movits!/Äppelknyckarjazz/04 Fast tvärtom.test.bin Five Iron Frenzy/Our Newest Album Ever!/11 Oh, Canada.test.bin 311/Don’t Tread On Me/02 Thank Your Lucky Stars.test.bin Movits!/Äppelknyckarjazz/01 Ta på dig dansskorna.test.bin Trans-Siberian Orchestra/Christmas Eve and Other Stories/05 The Silent Nutcracker (Instrumental).test.bin Apt° Core/2/01 No Such Thing As Time.test.bin Trans-Siberian Orchestra/Christmas Eve and Other Stories/06 A Mad Russian’s Christmas (Instrumental).test.bin 311/Don’t Tread On Me/11 There’s Always An Excuse.test.bin Five Iron Frenzy/Our Newest Album Ever!/01 Handbook for the Sellout.test.bin 311/Don’t Tread On Me/01 Don’t Tread On Me.test.bin Movits!/Äppelknyckarjazz/08 Tom Jones.test.bin Parov Stelar/Coco Pt.1/01 Coco (Featuring Lilja Bloom).test.bin Parov Stelar/Coco Pt.2/01 The Mojo Radio Gang(Radio Ver.).test.bin 311/Don’t Tread On Me/09 Whiskey & Wine.test.bin Parov Stelar/Coco Pt.2/02 Ragtime Cat(Featuring Lilja Bloom).test.bin Movits!/Äppelknyckarjazz/11 2 dollar på fickan.test.bin

status: 0 ignored, 0 errors, 39 pushed, 0 unchanged in 5.383501s

palladium:local spydez$ touch bar.txt palladium:local spydez$ echo foo > foo.txt

palladium:local spydez$ veb status veb repository at ~/test/scratch/local

Changed files:

bar.txt

modified on (2012-05-21 23:31:47 -0500 CDT)

foo.txt

filesize increased 4.00B (0.00B -> 4.00B)
modified on (2012-05-21 23:32:02 -0500 CDT)

MAKE SURE CHANGED FILES ARE THINGS YOU’VE ACTUALLY CHANGED (use ‘veb fix <file>’ if a file has been corrupted in this repository) (use ‘veb push’, ‘veb pull’, or ‘veb sync’ to commit changed/new files)

summary: 0 new, 2 changed (2.647ms)

palladium:local spydez$ veb push veb repository at ~/test/scratch/local

use ‘veb status’ to check new/changed files use ‘veb commit’ to add new/changed files to repository

LOCAL ignored files:

bar.txt foo.txt

status: 2 ignored, 0 errors, 0 pushed, 37 unchanged in 10.196ms

palladium:local spydez$ veb commit veb repository at ~/test/scratch/local

Committed files:

bar.txt foo.txt

summary: 2 commits, 0 errors in 3.222ms

palladium:local spydez$ veb push veb repository at ~/test/scratch/local

Pushed files:

foo.txt

status: 0 ignored, 0 errors, 1 pushed, 38 unchanged in 4.626ms

SLOC

Cheesy version $ wc -l `find . -iname “*.go”`
non-blank lines version find . -type f -iname “*.go” -print0 | xargs -0 cat | sed ‘/^\s*$/d’ | wc -l
[2012-05-21 Mon 14:28] palladium:src spydez$ find . -type f -iname “*.go” -print0 | xargs -0 cat | sed ‘/^\s*$/d’ | wc -l 1260

Files

project.org

Latest commit

History

project.org

File metadata and controls

Go links

Mysteries of Go

CMDs

users.txt

Costs

TODOs

line numbers (linum) in Go-mode buffers

Is fill-column not default 80 in my dotfiles?!

install HG for go-get

update Xcode for MacPorts for HG

TASKS

000: Go setup

001: walk dir tree, print out

002: SHA-256 vs SHA1

003: Hash file, print out result/time/etc

004: walk & hash

005: time stuff

006: goroutines

007: meta-file format

008: index file

009: xsums file

010: log [3/5]

011: parallize

012: bash scripts for testing [4/8]

013: !!! use channels of IndexEntries

014: commands [2/2]

init

status

verify [2/2]

commit

remote

push

help

fix

commit again

pull/sync

015: put globals somewhere global

016: clean deleted files out of index

017: FINALS [2/5]

999: update commands: remote

999: update commands: push

999: nice

999: future features

example

SLOC