tag:blogger.com,1999:blog-72143663695889847302024-03-07T20:55:48.528-08:00Insufficiently RandomThe lonely musings of a loosely connected software developer.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.comBlogger92125tag:blogger.com,1999:blog-7214366369588984730.post-68747378799300716682010-05-14T07:24:00.000-07:002010-05-14T08:20:09.725-07:00Don't Assume You Know What's BestYesterday I think we finally found the cause of <a href="http://code.google.com/p/gerrit/">Gerrit Code Review</a> <a href="http://code.google.com/p/gerrit/issues/detail?id=390">issue 390</a> and <a href="http://www.eclipse.org/jgit/">JGit</a> <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=308945">bug 308945</a>. In issue 390 a long-running Gerrit Code Review daemon suddenly loses access to objects in one or more Git repositories. The daemon's error log shows the server cannot find a commit, but <span class="Apple-style-span" style="font-family:'courier new';">git cat-file</span> on the command line is able to read the same commit without errors. A restart of the daemon JVM always corrects the problem. So we knew the problem had to be data corruption within JGit's in-memory caches.<div><br /></div><div>All along I've been looking for some sort of data corruption in the list of known pack files. The list is managed using volatiles and <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/atomic/AtomicReference.html">AtomicReferences</a>, and does most atomic operations itself, rather than building upon the data structures offered by the <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/package-summary.html">java.util.concurrent</a> package. Since I'm not <a href="http://gee.cs.oswego.edu/dl/index.html">Doug Lea</a>, its entirely possible that there is a race or unsafe write within this code. Fortunately, this code appears to be OK. I've gone over it dozens of times and cannot find a logic fault.</div><div><br /></div><div>Then along comes bug 308945, where JGit is reading from a closed pack file.</div><div><br /></div><div>JGit opens each pack file once using a <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/io/RandomAccessFile.html">RandomAccessFile</a>, and then uses the NIO API <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/nio/channels/FileChannel.html#read(java.nio.ByteBuffer, long)">read(ByteBuffer, long)</a> to execute a thread-safe <a href="http://opengroup.org/onlinepubs/007908799/xsh/pread.html">pread(2)</a> system call anytime it needs data from the file. This allows JGit to reuse the same file descriptor across multiple concurrent threads, reducing the number of times that it needs to check the pack file's header and footer. It also minimizes the number of open file descriptors required to service a given traffic load. To prevent the file from being closed by one thread while its being read by another, JGit keeps its own internal 'in-use' counter for each file, and only closes the file descriptor when this counter drops to 0. In theory, this should work well.</div><div><br /></div><div>Right until we mixed in <a href="http://mina.apache.org/sshd/">MINA SSHD</a>, a pure Java SSH server. When a client disconnects unexpectedly, MINA appears to be sending an <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Thread.html#interrupt()">interrupt</a> to the thread that last read data from the client connection. If that thread is currently inside of read(ByteBuffer,long), the read is interrupted, the file descriptor is closed, and an exception is thrown to the caller.</div><div><br /></div><div>Wait, what? The <i>file descriptor is closed</i>?</div><div><br /></div><div>And therein lies the bug. When I selected read(ByteBuffer,long) and this file descriptor reuse strategy for JGit, I failed to notice the documentation for throws <a href="http://java.sun.com/j2se/1.5.0/docs/api/java/nio/channels/ClosedByInterruptException.html">ClosedByInterruptException</a>. That oversight on my part lead to the misuse of an API, and an ugly race condition.</div><div><br /></div><div>When MINA SSHD interrupts a working JGit thread at the right place, the current pack file gets closed, but JGit's in-use counter thinks its still open. Subsequent attempts to access that file all fail, because its closed. When JGit encounters a pack file that is failing to perform IO as expected, it removes the file from the in-memory pack file list, but leaves it alone on disk. JGit never picks up the pack file again, as the pack file list is only updated when the modification time on the GIT_DIR/objects/pack directory changes. Without the pack file in the list, its contained objects cannot be found, and they appear to just vanish from the repository.</div><div><br /></div><div>I don't know what possessed the people who worked on <a href="http://jcp.org/en/jsr/detail?id=051">JSR 51</a> (New I/O APIs for the Java<sup><span style="font-size:-2;">TM</span></sup> Platform) to think that closing a file descriptor automatically during an interrupt was a good idea. RandomAccessFile's own read method doesn't do this, but its associated FileChannel does. In my opinion, they might as well have just invoked a setuid root copy of <a href="http://linux.die.net/man/8/halt">/sbin/halt</a> and powered off the host computer.</div>Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com7tag:blogger.com,1999:blog-7214366369588984730.post-55148704842659996972010-04-28T07:30:00.000-07:002010-04-28T07:48:14.341-07:00Why am I surprised when things work?I recently purchased a <a href="http://www.fujitsu.com/us/services/computing/peripherals/scanners/scansnap/scansnap-s1500m.html">Fujitsu ScanSnap S1500M</a>. This isn't interesting, its a scanner. You plug it into your computer, and its supposed to make picture files from paper. Yay. We've had scanners for ages. Its not blog worthy.<div><br /></div><div><b>What shocked me was, the damn thing does exactly what it says on the tin.</b></div><div><br /></div><div>Load its sheet feeder up with paper, plug it into the computer's USB port. Push the only "Scan" button on the front. Next thing you know, there is a folder full of sequentially numbered JPEG files. It automatically detects the length of the paper. It scans double-sided at the same speed it scans single-sided. It automatically drops back sides which are completely blank. Pages narrower than 8.5" are correctly detected and scanned with a narrower image width. It goes through 20 pages per minute. That's fast enough that its done before you realize its started.</div><div><br /></div><div>I realized after scanning several hundred pages in just a few minutes that very few things I purchase these days "just work". Most products still require a lot of tinkering from the user, or are still so complex that you need an advanced degree to operate them. This scanner, well, anyone's cat could use it. Just tap that scan button.</div><div><br /></div><div>Most products require you to purchase additional stuff, e.g. cables, to get them to work. Fujitsu actually included a USB cable in the box. Just unpack, plug in, and go. Its hard to argue with that. Even my <a href="http://www.tivo.com/">HD TiVo</a> was harder to get setup and going.</div><div><br /></div><div>To organize that directory of image files, I started using Brad Fitzpatrick's <a href="http://github.com/bradfitz/scanningcabinet">scanningcabinet</a> application. Though I did make a few changes in <a href="http://github.com/spearce/scanningcabinet">my own scanningcabinet fork on GitHub</a>. Now if only <a href="http://code.google.com/appengine/">Google AppEngine</a> supported <a href="http://googleappengine.blogspot.com/2010/04/making-your-app-searchable-using-self.html">full text search better</a>...</div>Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com3tag:blogger.com,1999:blog-7214366369588984730.post-71407404293523761262010-04-23T10:43:00.000-07:002010-04-23T10:49:04.808-07:00Gerrit Code Review on FLOSS WeeklyOn Wednesday I recorded a netcast for <a href="http://twit.tv/floss">FLOSS Weekly</a> with Randal Schwartz and Randi Harper about <a href="http://code.google.com/p/gerrit/">Gerrit Code Review</a>, <a href="http://www.eclipse.org/jgit/">JGit</a>, <a href="http://www.eclipse.org/egit/">EGit</a>, and <a href="http://git-scm.com/">Git</a> in general. The video and audio versions of the netcast are <a href="http://twit.tv/floss118">now available</a>.<div><br /></div><div>It was fun recording the show. I don't usually do these sorts of things, I find talking to a laptop somewhat challenging conceptually. Its just a thing sitting there, and it doesn't talk back. You can't see your audience's reactions to your words. I guess that's why I never got into radio, I couldn't sit and talk to a wall for four hours a day, every day. I definitely prefer getting up on stage and giving a talk in person.</div>Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-41363382318040450922010-04-12T14:17:00.000-07:002010-04-12T14:22:42.626-07:00Pre-testing commits with GitThe awesome folks who work on <a href="http://hudson-ci.org/">Hudson CI</a> have finally brought us <a href="http://blog.hudson-ci.org/content/pre-tested-commits-git">pre-tested commits</a> with <a href="http://code.google.com/p/gerrit/">Gerrit Code Review</a>. Their solution of watching everything under <span class="Apple-style-span" style="font-family:'courier new';">refs/changes/</span> is a bit brute-force, but its an amazing first start, because Hudson can "vote" on the change and prevent it from being submitted if the build failed.<div><br /></div><div>A few years ago I started a similar sort of thing for Git. Its carried in the <a href="http://git.kernel.org/?p=git/git.git;a=tree;f=contrib/continuous;hb=HEAD">contrib/continuous</a> directory of the <a href="http://git.kernel.org/?p=git/git.git;a=summary">git.git source code distribution</a>. But this whole Hudson-Gerrit integration is way better, because it lets you catch the failure before its submitted to your development branches.</div>Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-63828405618375296002010-04-07T14:02:00.001-07:002010-04-07T14:08:03.057-07:00Git is moving...Cedric recently wrote an interesting post on his blog, <a href="http://beust.com/weblog/2010/04/06/git-for-the-nervous-developer/">Git for the nervous developer</a>. Unlike a lot of the other blogs out there, he approached Git from the kicking and screaming angle, where he was already comfortable with another VCS and was forced to switch to Git for his day-job. Its an interesting perspective that he has, how he has found some sort of happiness with the tool he didn't choose to use.<div><br /></div><div>Today I also gave a talk on Git, JGit, EGit and Gerrit Code Review at the <a href="http://www.sonatype.com/about/in-the-news/20090915">Sonatype Maven Meetup</a> in Philadelphia. The talk was really well attended, according to Jason van Zyl, everyone chose my talk during the first time slot of the day. Most of the audience is apparently from the financial services sector, so they are a bit behind the bleeding edge of the open source VCS curve, but they were aware of Git and asking some really great questions about its capabilities. I'm glad I went, maybe we'll see some wider adoption of Git outside of the more usual open source communities.</div>Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-69456800860895660762010-03-29T17:08:00.000-07:002010-03-30T08:35:08.031-07:00JGit 0.7.1We finally managed to release <a href="http://dev.eclipse.org/mhonarc/lists/jgit-dev/msg00233.html">0.7.1</a> of <a href="http://www.eclipse.org/jgit/">JGit</a>, through the Eclipse Foundation's incubation process. Unfortunately we have yet to figure out how to get our Hudson CI server to produce a Maven update site and make that available through the download farm used by Eclipse projects. So we have yet to get an official Maven repository published.<div><br /></div><div>But Eclipse users can install <a href="http://www.eclipse.org/egit/">EGit</a> 0.7.1 through the official P2 update <a href="http://download.eclipse.org/egit/updates">site, http://download.eclipse.org/egit/updates</a>.</div><div><br /></div><div><b>[Update] </b>We now have an official <a href="http://www.eclipse.org/jgit/download/">JGit Maven site</a>.</div>Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-70351919561424517752010-03-27T17:46:00.000-07:002010-03-27T19:08:32.144-07:00Can't beat the cloudI have decided it is time to stop running my own web server just for my silly little blog. And that was about the only reason I'm still renting a virtual server from <a href="http://www.slicehost.com/">Slicehost</a> (err <a href="http://www.rackspace.com/">Rackspace</a>). At $20/month it just doesn't make much sense anymore.<div><br /></div><div>So I've moved the blog onto <a href="http://www.blogger.com/">Blogger</a>, static files onto <a href="http://aws.amazon.com/s3/">AWS S3</a>, and some small URL redirection glue onto <a href="http://code.google.com/appengine/">Google AppEngine</a>. With free DNS hosting provided by my registrar, free blog hosting at Blogger, and free redirection glue on AppEngine, I can cut my costs by nearly $20/month. The only real cost is the static files on S3, which is just a couple of tiny images. I fortunately have never had a very media rich site. Given S3's prices, this is pennies/month.</div><div><br /></div><div>The domain's email was moved months ago onto <a href="https://www.google.com/a/">Google Apps for Your Domain</a>. I just couldn't keep <a href="http://spamassassin.apache.org/">SpamAssassin</a> running with sufficiently up-to-date rules on a tiny virtual server with only 256 MB of memory allocated to it. Fortunately, Gmail works great over IMAP and direct SMTP. And I do like having the fast web based search every once in a while.</div><div><br /></div><div>This means the <a href="/2009/12/moving-my-git-repositories.html">git.spearce.org experiment</a> will be going away soon. Most likely I'll keep my Git repositories at <a href="http://github.com/spearce">GitHub</a>, or <a href="http://repo.or.cz/">repo.or.cz</a>. Fortunately these are small and generally just mirrors of open source projects whose primary repository lives elsewhere.</div>Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-40194760689659521022010-02-10T03:47:00.000-08:002010-03-27T14:26:41.837-07:00The Eclipse.org JGit follies continue...Another day. Another compliant from me about running a project at Eclipse.org. This time it wound up in the <a href="http://dev.eclipse.org/mhonarc/lists/jgit-dev">jgit-dev mailing list archives</a>, as replies to <a href="http://dev.eclipse.org/mhonarc/lists/jgit-dev/msg00103.html">a thread</a> that I think started from my blog post on <a href="http://www.spearce.org/2010/02/the-tragedy-of-eclipse-org.html">the tragedy of Eclipse</a>.<br/><br/>Instead of reposting the whole thing, I'll just point to my two messages in context:<br/><br/><a href="http://dev.eclipse.org/mhonarc/lists/jgit-dev/msg00109.html">why do I need to spend my time on this crap?</a><br/><a href="http://dev.eclipse.org/mhonarc/lists/jgit-dev/msg00111.html">why is the new file header what it is?</a>Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com2tag:blogger.com,1999:blog-7214366369588984730.post-14377378429186666652010-02-08T13:28:00.000-08:002010-03-27T14:26:39.976-07:00The tragedy of Eclipse.orgI've probably posted something about this before. But I'm really getting fed up with the <a href="http://www.eclipse.org/projects/dev_process/development_process.php">Eclipse Development Process</a>. Its a frelling nightmare for a committer to work with. I'm really starting to regret moving JGit there.<br/><br/>Right now, if I have X hours to work on a project, I seem to be averaging what feels like X/2 hours in paperwork and other "important steps" of the development process. None of which have helped my project to ship higher quality, or more feature complete code. Which means either my or my employer's time is being wasted. I don't have time to waste when I have 108 bugs open in <a href="http://code.google.com/p/gerrit/">Gerrit Code Review</a>, and 64 bugs open in <a href="http://www.eclipse.org/EGit">EGit</a> and <a href="http://www.eclipse.org/jgit/">JGit</a>.<br/><br/>Based on a private email chain I'm having with the Eclipse IP review team, it looks like the initial EGit code contribution was bungled not just by myself, but also by the foundation's IP review process. Which means I probably have to run EGit back through IP review, almost from scratch. But only after I write a script to datamine contributors out of the <a href="http://repo.or.cz/w/egit.git/shortlog/refs/heads/historical/pre-eclipse">old EGit history</a> and inject a complete, per-file <code>git short-log</code> into each file header. Its a good thing I have an <a href="http://git-scm.com/">awesome version control system like Git</a> to keep these records for me. Too bad nobody else on the planet can use it to obtain information they might want to know about our source code. I guess running software to read information about a file is too scary for some individuals. So I have to do it for them. Now, and for every change we make in the future. Yay. :-(<br/><br/>The astute reader may notice in that above paragraph, "private email chain" doesn't jive with other publications from the Eclipse Foundation demanding that projects be run in an open and transparent manner (see how do I start a project on <a href="http://www.eclipse.org/home/newcomers.php">Eclipse Newcomers</a>). I really do feel like JGit is a less open project now that it has moved to Eclipse.org. Conversations with the Eclipse IP team about the legal status of any contribution is always discussed by private email. These things never make it to the project mailing list. The IPzilla database is closed to everyone but committers. There are backroom deals going on about what our file headers should look like in order to sufficiently convey that the source code is under the <a href="http://www.opensource.org/licenses/bsd-license.php">new-style BSD</a>. The discussion that led to the approval of the <a href="http://www.eclipse.org/egit/iplog/v0.7.0.pdf">EGit IP log for 0.7.0</a>, approved despite what appears to be an error in the initial review, also happened by private email.<br/><br/>It took a significant amount of effort on my part to even get JGit hosted at Eclipse.org. Originally, the new-style BSD license wasn't permissible for a hosted project, and I had to seek a special exemption from the Eclipse Board of Directors. A process that required significant backroom conversations, over at least 6 months. Again, not exactly open. The only reason I think I haven't pulled the project back is because of the huge initial investment I've already made in this.<br/><br/>Maybe JGit and EGit are just unique projects. But in my experience, I am not a unique snowflake, and neither is my work. I'm not as special as I might seem at first glance.<br/><br/>I wouldn't be surprised if I've lost at least 2 days every month to paperwork. That's about 30 days, or 1.5 person-months since the project really started this move in January 2009. 1/12 of my time over the past year has just gone to catering to the Eclipse development process. Food for thought. Join Eclipse... make sure you pick up at least 1/12 of another full-time developer just to deal with the red-tape.<br/><br/>The part that really troubles me with the red-tape isn't so much that it is there, but that committers bear the brunt of the effort, while large corporations that are <a href="http://www.eclipse.org/membership/showMembersWithTag.php?TagID=strategic">strategic members</a> reap the benefits of having a concise change history listed inside of each source code file, or knowing that every contributor who ever touched this source code has been <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=300397#c4">grilled in detail on a bug tracker</a>.<br/><br/>So back to my post title. The real tragedy is, these corporations who sell commercial products based on top of Eclipse.org distributions are pushing not just the open source development work, but also a whole ton of onerous legal and reporting constraints back onto their project committers. Its enough to make this committer start to reconsider things. I wish I had been using a time clock this past year, to accurately record how many days the Eclipse development process has robbed me of since the start of all of this. It feels significant enough that if I went to my manager with it, I think he'd go ballistic.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com12tag:blogger.com,1999:blog-7214366369588984730.post-43402127208673233212010-02-04T06:04:00.000-08:002010-03-27T14:26:39.504-07:00Why commit messages matterSome folks wonder why I want longer, detailed commit messages in a project. Often other people claim "Fix the frobinator bug when it frobs too slow" might be sufficiently detailed to cover a change. But its usually not.<br/><a name='more'></a><br/>As you explored the issue and tried to understand the problem, you filled your head up with important details about how the frobinator works, what a frob even is, what a slow frob looks like, and why a slow frob shouldn't be permitted in this context. All of this information is necessary for you to understand the problem and code a patch that resolves it. Moreover, if this detail wasn't necessary for you to code the patch, you wouldn't have had the slow frobbing in the first place. It would have been fairly obvious at the time of original development.<br/><br/>Commit messages, when combined with a powerful blame engine in your version control, can give you really powerful insight into what you were thinking at the time. This can be incredibly handy when someone asks a question later.<br/><br/>Yesterday, <a href="http://gitster.livejournal.com/">Junio Hamano</a>, git maintainer extraordinaire, asked me why <a href="http://git.spearce.org/?p=git-gui.git;a=summary">git-gui</a> implements its own clone function. When I wrote this code, it must have been really obvious to me why it needed to reimplement the same logic as <a href="http://www.kernel.org/pub/software/scm/git/docs/git-clone.html"><code>git clone</code></a>. But I wrote it back in 2007. I've done a ton of things since then. There's no way I can remember what I was doing, or why I was doing it. I do however remember thinking, "this code is done, it works, I'll never have to look at or think about it again". Famous last words.<br/><br/>When Junio asked this question... I honestly couldn't remember what I was doing. I'm usually somewhat against reinventing the wheel, and I try to avoid rewriting something unless I seem to have a good reason for it. So I really was looking at his question saying, "yea, why did I do that there...".<br/><br/>Fortunately, I write fairly detailed commit messages, and <a href="http://www.kernel.org/pub/software/scm/git/docs/git-blame.html"><code>git blame</code></a> is an incredible tool:<br/><br/><blockquote><pre><br/> $ git blame lib/choose_repository.tcl<br/> ...<br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 633) <br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 634) $o_cons start \<br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 635) [mc "Counting objects"] \<br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 636) [mc "buckets"]<br/> ...<br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 673) update<br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 674) <br/> ab08b363 (Shawn O. Pearce 2007-09-22 03:47:43 -0400 675) file mkdir [file join .git objects pack]<br/> ab08b363 (Shawn O. Pearce 2007-09-22 03:47:43 -0400 676) foreach i [glob -tails -nocomplain \<br/> ab08b363 (Shawn O. Pearce 2007-09-22 03:47:43 -0400 677) -directory [file join $objdir pack] *] {<br/> ab08b363 (Shawn O. Pearce 2007-09-22 03:47:43 -0400 678) lappend tolink [file join pack $i]<br/> ab08b363 (Shawn O. Pearce 2007-09-22 03:47:43 -0400 679) }<br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 680) $o_cons update [incr bcur] $bcnt<br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 681) update<br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 682) <br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 683) foreach i $buckets {<br/> ab08b363 (Shawn O. Pearce 2007-09-22 03:47:43 -0400 684) file mkdir [file join .git objects $i]<br/> ab08b363 (Shawn O. Pearce 2007-09-22 03:47:43 -0400 685) foreach j [glob -tails -nocomplain \<br/> ab08b363 (Shawn O. Pearce 2007-09-22 03:47:43 -0400 686) -directory [file join $objdir $i] *] {<br/> ab08b363 (Shawn O. Pearce 2007-09-22 03:47:43 -0400 687) lappend tolink [file join $i $j]<br/> ab08b363 (Shawn O. Pearce 2007-09-22 03:47:43 -0400 688) }<br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 689) $o_cons update [incr bcur] $bcnt<br/> 81d4d3dd (Shawn O. Pearce 2007-09-24 08:40:44 -0400 690) update<br/> ab08b363 (Shawn O. Pearce 2007-09-22 03:47:43 -0400 691) }<br/></pre></blockquote><br/><br/>It would seem that <a href="http://git.spearce.org/?p=git-gui.git;a=commit;h=81d4d3dddc5e96aea45a2623c9b1840491348b92"><code>81d4d3dd</code></a>, and <a href="http://git.spearce.org/?p=git-gui.git;a=commit;h=ab08b3630414dfb867825c4a5828438e1c69199d"><code>ab08b363</code></a> are commits adding code to do a clone.<br/><br/><blockquote><pre><br/> $ git show 81d4d3dd<br/> commit 81d4d3dddc5e96aea45a2623c9b1840491348b92<br/> Author: Shawn O. Pearce <spearce <at> spearce.org><br/> Date: Mon Sep 24 08:40:44 2007 -0400<br/><br/> git-gui: Keep the UI responsive while counting objects in clone<br/><br/> If we are doing a "standard" clone by way of hardlinking the<br/> objects (or copying them if hardlinks are not available) the<br/> UI can freeze up for a good few seconds while Tcl scans all<br/> of the object directories. This is espeically noticed on a<br/> Windows system when you are working off network shares and<br/> need to wait for both the NT overheads and the network.<br/><br/> We now show a progress bar as we count the objects and build<br/> our list of things to copy. This keeps the user amused and<br/> also makes sure we run the Tk event loop often enough that<br/> the window can still be dragged around the desktop.<br/><br/> Signed-off-by: Shawn O. Pearce <spearce <at> spearce.org><br/><br/> $ git show ab08b363<br/> commit ab08b3630414dfb867825c4a5828438e1c69199d<br/> Author: Shawn O. Pearce <spearce <at> spearce.org><br/> Date: Sat Sep 22 03:47:43 2007 -0400<br/><br/> git-gui: Allow users to choose/create/clone a repository<br/> …<br/> Rather than relying on the git-clone Porcelain that ships with<br/> git we build the new repository ourselves and then obtain content<br/> by git-fetch. This technique simplifies the entire clone process<br/> to roughly: `git init && git fetch && git pull`. Today we use<br/> three passes with git-fetch; the first pass gets us the bulk of<br/> the objects and the branches, the second pass gets us the tags,<br/> and the final pass gets us the current value of HEAD to initialize<br/> the default branch.<br/><br/> If the source repository is on the local disk we try to use a<br/> hardlink to connect the objects into the new clone as this can<br/> be many times faster than copying the objects or packing them and<br/> passing the data through a pipe to index-pack. Unlike git-clone<br/> we stick to pure Tcl [file link -hard] operation thus avoiding the<br/> need to fork a cpio process to setup the hardlinks. If hardlinks<br/> do not appear to be supported (e.g. filesystem doesn't allow them or<br/> we are crossing filesystem boundaries) we use file copying instead.<br/><br/> Signed-off-by: Shawn O. Pearce <spearce <at> spearce.org><br/></pre></blockquote><br/><br/>So 30 seconds after being asked, I've managed to remember this was mostly about git-gui on Windows, where Cygwin can be pretty slow for file operations, and hardlinks are available on NTFS if your application knows how to make them. By doing the clone logic within Tcl, which is a native Win32 application, we can bypass Cygwin overheads, including the need to fork and execute a bunch of commands from the <code>git-clone.sh</code> shell script. Because, back in 2007, git-clone was still just a shell script.<br/><br/>In hindsight, that paragraph above should also be in the commit messages. And I probably should have ported git clone to C instead. Its C now, but not because of my efforts. And now git-gui maybe should just call it. It would have made git-gui a whole lot smaller.<br/><br/>You can follow the <a href="http://thread.gmane.org/gmane.comp.version-control.git/138612/focus=138874">rest of the thread</a>.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com5tag:blogger.com,1999:blog-7214366369588984730.post-11926815811122233992010-01-04T12:28:00.000-08:002010-03-27T14:26:39.494-07:00How class names can go horribly wrongSomehow I found myself writing this in a JGit test case:<br/><blockquote><br/><pre><br/>assertTrue("isa TransportHttp", t instanceof TransportHttp);<br/>assertTrue("isa HttpTransport", t instanceof HttpTransport);<br/></pre><br/></blockquote><br/>What is wrong with me...Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-51068629128803775382009-12-15T11:57:00.000-08:002010-03-27T14:26:39.485-07:00Eclipse is the new RPM hellRemember back when RedHat was the best Linux distribution? If not, let me remind you of <a href="http://www.germane-software.com/~ser/Files/Essays/RPM_Hell.html">RPM hell</a>. A situation where you can't install Foo 1.X, because it needs Bar 2.3.Y, but you also have Zidget 8.Z installed and that needs Bar 2.2.Y. Long story short, you can have either Foo or Zidget on your system, but not both.<br/><br/>Today I just tried to install the <a href="http://www.eclipse.org/tptp/">Eclipse Test & Performance Tools</a>, so I could try to find out why <a href="http://mina.apache.org/sshd/">Apache MINA SSHD</a> has such poor throughput during uploads into the server. Unfortunately I can't run version 4.5 because it <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=240677#c2">depends on a decade old version of libstdc++</a>. But I can't install version 4.6, which supposedly has a newer linkage, because the 4.6 requires SWT 3.4.0. But I'm running Eclipse 3.4.2, which apparently does not have a new enough SWT.<br/><br/>Folks. Seriously?<br/><br/>The only major consumer of SWT that really matters is the Eclipse SDK. And the SDK platform version numbers don't match SWT version numbers. And the test and performance tools require a decade old shared library which isn't even distributed with <a href="http://releases.ubuntu.com/hardy/">Ubuntu Hardy</a>.<br/><br/>I guess since its Java, its OK to repeat decade old mistakes, because its in a different programming language.<br/><br/>:-(Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-22893658555302251492009-12-08T19:59:00.000-08:002010-03-27T14:26:39.472-07:00My ancient history and the art of softwareThis week I'm traveling. I'm in Miami for the 2 day Eclipse board of directors meeting. When I'm forced to fly, which I usually try to avoid doing, I tend to take a stack of books with me to try and pass the time on the airplane. Unfortunately, airline flight speeds haven't quite caught up with Moore's law as it relates to the power draw of modern laptops. Nor has the airline seat space kept up with the size of my 15" PowerBook and my increasing girth. So books it is.<br/><br/>So I'm currently reading Peter Seibel's <a href="http://www.codersatwork.com/">Coders at Work</a>. Tonight, while reading through Brendan Eich's interview and his history at Netscape, it reminded me of my own development history around the same time period. I really don't talk about my past very much, so before I get to be too old to remember it, I might as well write some of it down. :-)<br/><br/><a name='more'></a>During the mid-to-late 90's I was still in high school, but was working part-time in the afternoons after school as a software developer (err, script monkey) for a now defunct website and ISP called <a href="http://web.archive.org/web/19961106235534/http://www.injersey.com/">INJersey</a>. INJersey was the aspiring, and perhaps too early for its time, online arm of the local daily print paper, the <a href="http://www.app.com/">Asbury Park Press</a>. Somehow its owners realized this Internet thing was worth investing in, before a lot of users figured it out and actually got online.<br/><br/>Back then, we didn't have <a href="http://httpd.apache.org/">Apache</a> and all of its module glory. We had a pile of patch files you had to apply to NSCA httpd to get <strong><a href="http://httpd.apache.org/docs/1.3/misc/FAQ.html#name">A PA</a></strong><a href="http://httpd.apache.org/docs/1.3/misc/FAQ.html#name">t</a><strong><a href="http://httpd.apache.org/docs/1.3/misc/FAQ.html#name">CH</a></strong><a href="http://httpd.apache.org/docs/1.3/misc/FAQ.html#name">y server</a>. We didn't have FastCGI, we had plain old CGI scripts, often written in Perl 4, where <code>&article'render()</code> was an actual function call and not some syntax error brought on by a lack of caffeine. We didn't even have JavaScript or animated images. <a href-"http://www.google.com/search?q=netscape+blink+tag"><blink></a> was as good as it got.<br/><br/>My job back then was really simple. Take Microsoft Office documents from writers who couldn't be bothered to learn HTML, and put them online in HTML. I wrote a lot of Perl and AppleScript to rip the stuff apart and put it back together again as plain HTML files that we could serve to web users. This was out of shear laziness, the <a href="http://c2.com/cgi/wiki?LazinessImpatienceHubris">first great virtue of a programmer</a>. INJersey hired me not as a software developer, but as a data entry monkey to copy and paste the text from Office into an HTML document, and stick in the <td> or <b> tag when necessary. I quickly grew bored with that task and found it much more fun to write scripts to do my job for me, while I explored the wonders of a <a href="http://en.wikipedia.org/wiki/T-carrier">T1</a> internet connection.<br/><br/>My managers quickly realized I was able to do more than just copy and paste text with a computer, so they started giving me simple programming assignments. When Netscape 2.0 launched, one of our real programmers figured out we could do <a href="http://oreilly.com/openbook/cgi/ch11_01.html">server-push based animated images</a>. This is one of those monstrously stupid ideas that I'm glad has died on the web. I'm quite happy that I actually can't come up with a great reference link for it anymore. Anyway, INJersey's website was just awesome one day, because we had animated images, and others didn't.<br/><br/>About this time one of my managers saw this website called JangaChat. It was a free online web based chat room system. No IRC client needed. No Java applets. No plugins. Just Netscape 2.0. Even better, they allowed HTML to be entered without escaping, so you could write fancy messages like "Bob, that was the best fish <b>ever</b>!" and actually have it come out in bold. Their site was more awesome than our animated images. We somehow had to out awesome them again.<br/><br/>Remember, this is like 1995. We just got <a href="http://www.antipope.org/charlie/journo/netscape.html">Netscape 2.0</a>, and <a href="http://inventors.about.com/od/jstartinventions/a/JavaScript.htm">Brendan Eich</a> had just unleased this JavaScript thing on the world. We didn't know what we could do with it... or the damage it could cause. Cross site scripting hadn't even been invented yet.<br/><br/>My managers gave me a simple task, create an INJersey version of JangaChat that we could run on our own servers, so our users could chat online in real time about whatever they felt like, without needing to first install an IRC client.<br/><br/>At the time, I only really knew Perl, and was only self-taught at that. So I wrote the first version as a Perl CGI. I remember abusing the server-based multipart push feature we used for image animation to allow the Perl CGI to stream new messages to the browser as they arrived in the chat room. JangaChat, and this system I worked on at INJersey, may have been some of the first uses of <a href="http://alex.dojotoolkit.org/2006/03/comet-low-latency-data-for-the-browser/">hanging GET</a>s.<br/><br/>Unfortunately, not only did I not know C, I also didn't know how to do proper interprocess communication on UNIX. So I implemented with what I did know: each chat room was assigned a local file. New messages were appended onto the end of the file using a POST CGI, and messages were read out to active users by their hanging GET CGIs, which were continuously trying to read the tail of the room's file. Over a decade later, I can't believe I was once foolish enough to believe this was a good idea.<br/><br/>We launched under the name <a href="http://web.archive.org/web/19980111133651/http://chat.injersey.com/">ChatterBox</a>. We quickly stole most of the users from JangaChat, plus picked up our own users, and our little Pentium 133 server could barely keep up with all of those Perl based hanging GET CGI processes demanding resources. Messages were also often truncated or otherwise badly mangled, as I had no file locking, and no way to ensure a full message was written before being read and sent to a browser.<br/><br/>Management really wanted this product to work, because the demo was flashy, and their local advertising customers were wowed by it. I rewrote the Perl code into C, but kept the basic design of individual hanging GET CGIs per user, and a single log file per chat room. Even in C, we still couldn't keep up with traffic. Somehow I convinced management that buying some "real server hardware", rather than our tiny single processor Pentium 133, would solve the scaling problems, and they went out and purchased a pair of <a href="http://en.wikipedia.org/wiki/SGI_Origin_200">SGI Origin 200</a> servers running Irix. Today of course, all 3 of you reading this are screaming "but its the software that wasn't scalable!". That's what a decade of learning gets you. :-)<br/><br/>So upon getting this shiny new server hardware, I start thinking about how I might have done things differently. I knew I couldn't rely on the (then pretty crappy and unscalable) dbm library for user data management, like I had with Perl. So I started poking around at things like <a href="http://www.hughes.com.au/products/msql/">mSQL</a>, <a href="http://www.postgresql.org/">PostgreSQL</a>, and <a href="http://www.mysql.com/">MySQL</a>. Back in 1996, mSQL was the best there was, but it was single threaded. PostgreSQL was pretty slow and barely ran, it was more of a research project than it was a production database engine. MySQL crashed more than it stayed up, severely lacked features compared to mSQL, and required nasty pthread libraries which weren't exactly standard or well supported on UNIX systems back then.<br/><br/>So I toyed with writing my own. As it turned out, Irix had a decent pthread implementation at the time. I actually ended up with a thread safe, balanced B*-tree that stored user record data in fixed size leaf nodes, and used an arbitrary byte sequence as the record key. To make the implementation easier on myself, I made every block of the file the same size (I think I used 2048 bytes, but I can't remember), and the parent that pointed to the block told you the type using the lower bits of the file offset. E.g. offset & 1 == 0 meant the block was another btree intermediate node, while offset & 1 == 1 meant it was an application data record.<br/><br/>Within the user data record, I didn't want to write all of the glue required for a formal DDL like a proper SQL server would support. After all, I just had to store a couple of values for each user account, like their email address, date of last login, and a handful of preference settings. So I basically did what <a href="http://code.google.com/apis/protocolbuffers/docs/overview.html">Google protocol buffers</a> does, and stored a flexible bag of key/value pairs in binary form. Schema changes were accomplished incrementally, as users were updated during normal application processing, rather than all up front when the software changed.<br/><br/>So, I solved my data storage problem by just rolling my own system. Looking back on it, its not too different from the approach Google takes with <a href="http://labs.google.com/papers/bigtable.html">BigTable</a>. Only they scale across machines by partitioning the key space, while I just tossed the entire key space onto a single node.<br/><br/>For the interactive web components, I finally got a clue and realized that the hanging GET CGI processes were my real scaling factor. They consumed too much of the system's resources per process. My local UNIX system administrator finally got around to telling me about sar and ps and how to use them to figure out that I was being a moron.<br/><br/>Since we still didn't have FastCGI, I wrote my own HTTP server. From scratch. In C. First and last time I've ever done that. Lesson learned. There's a reason I haven't been involved in the Apache HTTPd project. My scars still haven't healed.<br/><br/>Initially, my HTTP server was using pthreads, spawning a pthread per browser request. Which meant that my server used 1 thread per hanging GET. At first I thought this would be OK, threads were lightweight compared to a process, right? Until my local UNIX administrator brought out the clue stick and made me think about it for a few minutes. In the process-per-connection implementation we had very little allocated memory, just a kilobyte or so of global state, and the entire program executable was mapped as shared memory between all of the instances. The real memory cost was in the program's stack, which the OS had a minimum size on. With pthreads we didn't gain much resource savings over the process-per-connection approach because we still had the same per-connection state, and the same minimum thread stack size.<br/><br/>I started digging around the Irix manual pages, and one day found this nifty thing called <a href="http://linux.die.net/man/2/select">select()</a>. I wrote a quick simple server using it, and realized non-blocking asynchronous IO was the best thing since sliced bread. That night I completely tore apart the pthread based HTTP server and rewrote it as a non-blocking, asynchronous IO server.<br/><br/>By now I had completely abandoned the idea of the message distribution going through a local log file, and instead stored it in shared memory protected by pthread mutexes and condition variables. Threads posting messages into a chat room appended their message object onto a distribution queue and signaled one of the IO threads to start streaming the messages out using asynchronous IO to each connected browser.<br/><br/>Along the way I also had to write an HTML parser, because we wanted to allow messages like "Here is a <font color=red>red rose</font>" to render as intended by the author, but we didn't want to have unclosed tags like "<h1>Hi!" ruin the formatting for every subsequent message on the page. End users also figured out cross site scripting before we did, with plenty of "<script>window.close()</script>" nosense being pasted into rooms and closing hundreds of browsers at once. Yes, we learned about cross site scripting attacks the hard way. Back in 1996/97, nobody really thought about this sort of stuff.<br/><br/>We relaunched in the winter of 1997. The Wayback machine has a snapshot of the rewritten <a href="http://web.archive.org/web/19980111133651/http://chat.injersey.com/">homepage dated April 13, 1997</a>.<br/><br/>Of all of that though, the thing I'm still most excited about is the "non-Shockwave" version of the ChatterBox website. Back then we didn't have AJAX and XMLHttpRequest. Heck, we didn't even have iframe. We had straight up boring <frameset>. I rolled my own AJAX system from scratch using JavaScript, a hand rolled message queue, and a 1 pixel high frame in a frameset that performed POSTs driven by JavaScript. In very early 1997. Apparently I discovered parts of the web that we didn't see again until XMLHttpRequest was widely supported.<br/><br/>Unfortunately management pulled the plug a year or so later, and that neat little HTTP server with its pre-AJAX AJAX and pre-Google Talk hanging GETs disappeared into the sands of time.<br/><br/>So, tonight, while reading Brendan Eich's interview talking about building JavaScript in two weeks, I remembered just how much I learned that year in my part time job. I went from being a Perl script monkey who couldn't even compile a C program, to having written an asynchronous IO HTTP server from the ground up, mastered multi-threading, mutexes, condition variables, AJAX, cross site scripting, and a balanced binary tree implementation that bears a lot of resemblance to BigTable. All without taking any CS classes in high school.<br/><br/>Maybe that's why college, and my later jobs, were all so boring for me. I'd done all of it already. Maybe that's why I was less amazed than my peers when Google Maps launched. I didn't make something nearly as awesome, or as empowering to so many people around the world as Lars and the Maps team did. But I had certainly seen, worked with, and discovered much of today's web. In 1997.<br/><br/>What is <a href="http://www.whatwg.org/">HTML 5</a> going to bring us that we haven't discovered yet?Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-75509980668772581162009-12-07T19:19:00.000-08:002010-03-27T14:26:39.159-07:00Introduction to Gerrit Code ReviewYesterday R. Tyler Ballance (aka rtyler on <a href="irc://irc.freenode.net/%26git">#git</a>) started poking me about <a href="http://code.google.com/p/gerrit/">Gerrit Code Review</a>. One day later, he's writing an <a href="http://unethicalblogger.com/posts/2009/12/code_review_gerrit_mostly_visual_guide">amazing blog post</a> describing how to install the latest development version, and why it so awesome to use for team development. He even has screenshots! :-)<br/><br/>Thanks rtyler.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com2tag:blogger.com,1999:blog-7214366369588984730.post-87984801505039809502009-12-07T08:37:00.000-08:002010-03-27T14:26:39.149-07:00EGit in 2010?Mike just described some changes coming to Eclipse in 2010, <a href="http://dev.eclipse.org/blogs/mike/2009/12/07/project-community-enhancements-for-2010/">including Git for projects</a>. Yay!<br/><br/>However, he's right. EGit has to get better, fast. We need more contributors who know and love the SWT/JFace/Resource APIs and can crank out the UI improvements necessary to bring the Git team provider up to the same level as the CVS team provider.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-48871124534209879462009-12-05T09:01:00.000-08:002010-03-27T14:26:39.141-07:00Moving my git repositoriesI finally got around to creating <a href="http://git.spearce.org/">git.spearce.org</a>. I've been contributing to Git since <a href="http://git.kernel.org/?p=git/git.git;a=commit;h=772d8a3b63ff669c285edb8aff0c63b609614933">Feb 17 2006</a> and yet I couldn't be bothered to setup my own Git host for my repositories. For the past 3 years I've primarily leaned on Pasky's excellent <a href="http://repo.or.cz/">repo.or.cz</a> service. But I've always wanted a more permanent home for my projects.<br/><br/>Last week I threw <a href="http://spamassassin.apache.org/">SpamAssassin</a> off my domain's server, which meant I finally had virtual memory free to run <a href="http://www.kernel.org/pub/software/scm/git/docs/git-daemon.html"><code>git daemon</code></a>. So you can now find most of my projects at <a href="http://git.spearce.org/">git.spearce.org</a>, with proper git:// URLs available for efficient cloning. Maybe sometime later this month I'll get smart HTTP enabled as well.<br/><br/>I'll continue to update the repositories on repo.or.cz, but I'll primarily be using the ones hosted on git.spearce.org.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-85842860182771736532009-12-03T09:36:00.000-08:002010-03-27T14:26:39.132-07:00EGit at EclipseA few months ago we moved <a href="http://www.eclipse.org/egit/">EGit</a>, the Git team provider for Eclipse, over to the <a href="http://www.eclipse.org/org/">Eclipse Foundation</a>. Along the way we decided to try out some new development techniques, like taking advantage of my day-job project <a href="http://code.google.com/p/gerrit/">Gerrit Code Review</a> to help us discuss pending changes. This lead us down the road of not paying too much attention to the <a href="http://www.eclipse.org/projects/dev_process/ip-process-in-cartoons.php">Eclipse IP process</a>, and failing to tag all contributed patches with the +iplog flag in Bugzilla.<br/><br/>Fortunately Wayne Beaton helped us get Gerrit configured in a way that meets the foundation's IP process guidelines, and has <a href="http://dev.eclipse.org/blogs/wayne/2009/12/01/git-at-eclipse/">encouraged us to continue forward</a>. This is great, because it means we can rely on Git for attribution tracking, rather than Bugzilla.<br/><br/>Also, since the project moved homes we picked up 3 prolific contributors, and all of them have turned into committers on the project.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-91499355238061087302009-12-03T09:28:00.000-08:002010-03-27T14:26:39.125-07:00Been a while...Has it really been over a year since I last updated spearce.org?<br/><br/>Yup. It has.<br/><br/>A lot happens in a year. We moved. I got a new job. I got a lot of new projects. I started spending a lot of time working on <a href="http://git-scm.com/">Git</a>, or more precisely, <a href="http://www.eclipse.org/egit/">JGit</a>. And I let my amusing little corner of the web grow cobwebs and wither.<br/><br/>Yay.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-59677381332000782182008-07-18T08:23:00.000-07:002010-03-27T14:26:39.115-07:00Getting Giddy with GitRecently Johannes Schindelin and I participated in a podcast about Git's involvement in <a href="http://code.google.com/soc/">Google Summer of Code</a>. You can listen to the podcast on <a href="http://google-opensource.blogspot.com/2008/07/getting-giddy-with-git.html">the Google open source blog</a>.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-28166228569834493182008-07-08T17:06:00.000-07:002014-07-03T12:37:10.667-07:00Using jgit To Publish on Amazon S3Recent versions of <a href="http://eclipse.org/jgit/download/">jgit</a>, the 100% pure Java implementation of the <a href="http://git.or.cz/">Git version control system</a>, support fetch and push directly over <a href="http://aws.amazon.com/s3">Amazon S3</a> .<br />
<br />
It behaves like http push does in C git in that it is transparent to the end-user. Transparent client-side encryption can also be enabled, in case the repository data must be protected from the operators of S3.<br />
<a name='more'></a><br />
First you need to create a bucket using some sort of standard S3 tools. I used <a href="http://jets3t.s3.amazonaws.com/index.html">jets3t's cockpit</a> tool to create "gitney". A bucket may hold any number of repositories and acts as a root directory. It may also be a domain name if you want to use <a href="http://docs.amazonwebservices.com/AmazonS3/2006-03-01/VirtualHosting.html">S3 based virtual hosting</a>.<br />
<br />
Next you need to create a properties file containing your AWSAccessKeyId and AWSSecretAccessKey so that jgit can authenticate itself with the S3 service. Since the AWSSecretAccessKey should be maintained privately its a good idea to store this in a protected file within your home directory.<br />
<br />
<pre>
$ touch ~/.jgit_s3_public
$ chmod 600 ~/.jgit_s3_public
$ cat >>~/.jgit_s3_public
accesskey: AWSAccessKeyId
secretkey: AWSSecretAccessKey
acl: public
EOF</pre>
<br />
<br />
We also include <code>acl: public</code> so all objects (files) created by jgit through this configuration file are readable by anyone. The default (if not specified) is <code>acl: private</code>, making the objects readable only by yourself, and those who manage the S3 service.<br />
<br />
Next we configure the remote in Git and push to the S3 bucket:<br />
<pre>
$ git remote add s3 amazon-s3://.jgit_s3_public@gitney/projects/egit.git/
$ jgit push s3 refs/heads/master
$ jgit push --tags s3</pre>
<br />
<br />
Future updates are just as easy:<br />
<pre>
$ jgit push s3 refs/heads/master</pre>
<br />
(or)<br />
<pre>
$ git config --add remote.s3.push refs/heads/master
$ jgit push s3</pre>
<br />
Pushes are always incremental and consequently there is relatively little bandwidth usage during subsequent pushes.<br />
<br />
Our repository is now cloneable directly over HTTP (assuming we used <code>acl: public</code>):<br />
<pre>
$ git clone http://gitney.s3.amazonaws.com/projects/egit.git</pre>
<br />
<br />
A jgit amazon-s3 URL is organized as:<br />
<pre>
amazon-s3://$config@$bucket/$prefix
http://$bucket.s3.amazonaws.com/$prefix</pre>
<br />
where the three major components are:<br />
<ul><br />
<li><code>$config</code> is the name of the configuration properties file stored in <code>$GIT_DIR/$config</code> or <code>$HOME/$config</code> (searched for in that order).</li>
<br />
<li><code>$bucket</code> is the name of the Amazon S3 bucket holding the objects.</li>
<br />
<li><code>$prefix</code> is the prefix to apply to all objects (files) within this repository. It implicitly ends in "/". You may omit this portion of the URI if you want the bucket to contain only one repository.</li>
</ul>
<br />
This is something of an abuse of URI syntax as the traditional username field is holding the name of a file in either <code>$GIT_DIR</code> or <code>$HOME</code>, however it permits hiding the secret access key from prying eyes as well as supplies a way to carry more information (such as acl or encryption settings) than what can appear in a URI.<br />
<br />
Transparent client-side encryption for a repository stored on S3 can be enabled by adding a <code>password</code> to the properties file:<br />
<pre>
$ cp ~/.jgit_s3_public ~/.jgit_s3_private
$ echo password: Sup3rS3cr3t >>~/.jgit_s3_private</pre>
<br />
and using <code>.jgit_s3_private</code> in the $config field of an amazon-s3:// URL. The encryption algorithm can also be specified in property <code>crypto.algorithm</code>, which defaults to PBEWithMD5AndDES.<br />
<br />
The encryption format currently used by jgit matches the format used by jets3t (specifically format version 2), making it possible to download and decrypt a repository through cockpit in the event that jgit is not readily available.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com8tag:blogger.com,1999:blog-7214366369588984730.post-15271800567240921482008-06-26T16:59:00.000-07:002010-03-27T14:26:38.276-07:00New Server, New HomeIn honor of us moving from New York to California I have also moved the server that hosts spearce.org. If you are reading this blog post, welcome to my new home away from home on the web.<br/><br/>If you aren't reading this blog post, you may need to reconsider how you reached this location, since you can't see it.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-4692986631054677032007-07-26T15:59:00.000-07:002010-03-27T17:07:27.285-07:00Difficult gitk GraphsOn the Git mailing list I've talked about one of the repositories I develop on/maintain, as its graph in gitk is somewhat interesting. Today I took a couple of redacted screenshots from two of the interesting parts of the history.<br/><br/>The first image is from a set of octopus merges that occurred in the history. This probably would look better if we had just used <a href="http://www.kernel.org/pub/software/scm/git/docs/git-rebase.html"><code>git-rebase</code></a> to transplant the commits instead of merging them, but at the time the user who created these was still quite new to Git...<br/><br/><p style="margin-left: 40px"><a href="http://d.spearce.org/2007/07/wide-gitk.gif"><img alt="Many Branches and Octopus Merges" title="Many Branches and Octopus Merges" src="http://d.spearce.org/2007/07/wide-gitk-thumb.gif" /></a><br/><a href="http://d.spearce.org/2007/07/wide-gitk.gif"><em><font size="-1">(larger version)</font></em></a><br/><br/><a name='more'></a><br/><br/>What's even worse about that particular rendering is the branches on either side. This was taken with the following gitk preferences set:<br/><blockquote>Maximum graph width (lines): 80<br/>Maximum graph width (% width of pane): 90</blockquote><br/>Even with those settings, gitk still does not have enough space to show all of the active branches passing through this particular point in time (see the lines cut off on the upper right corner). In case you are wondering, yes, nearly all of those merged together later on. A few haven't yet.<br/>This second image was taken from a more recent point in the project's timeline, during the same gitk session, and with the same preferences. We've obviously reduced the number of active branches somewhat, and there is now space for the commit subject lines (which I had to redact) and tracking branch labels (also had to be redacted).<br/><br/><p style="margin-left: 40px"><a href="http://d.spearce.org/2007/07/ugly-gitk.gif"><img alt="Chained Merges" title="Chained Merges" src="http://d.spearce.org/2007/07/ugly-gitk-thumb.gif" /></a><br/><a href="http://d.spearce.org/2007/07/ugly-gitk.gif"><em><font size="-1">(larger version)</font></em></a><br/><br/>There is still an octopus here ("Cauterize prior batch m"), but this one was automatically generated by a script we developed and was not directly caused by a user. The script takes a single commit and merges it to all branches that share a common prefix. I don't have the repository on hand right now, but I think this particular octopus was created while we were merging one commit to ~80 branches. In this case 4 branches were already fully merged with a 5th, and we wanted to keep it that way after the new commit was merged to all 80 branches. To do that the script remerged these 4 by way of an octopus.<br/><br/>Before you can ask, no, this is not an imported repository. This was all created in git, using the core porcelain and <a href="http://repo.or.cz/w/git-gui.git/">git-gui</a>.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com2tag:blogger.com,1999:blog-7214366369588984730.post-78671405387185300262007-04-28T17:50:00.000-07:002010-03-27T14:26:37.622-07:00pg is for saleI have decided to no longer support pg, as I haven't used it myself in a very, very long time. It was a useful tool and learning vehicle for myself and a few others, but it just isn't nearly as good as core Git with topic branches. Or <a xhref="/category/projects/scm/git-gui/">git-gui</a>.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-52682368793487339702007-02-28T18:24:00.000-08:002010-03-27T17:06:27.436-07:00Git and Linux Repository GrowthI got curious about the growth rate for the git.git and linux-2.6.git repositories, so I wrote <a href="http://article.gmane.org/gmane.comp.version-control.git/41042"><code>git-statplot</code></a> to dump out object counts and sizes by earliest date entered. Plotting these with Gnuplot gave me some interesting results:<br/><br/><a href="http://d.spearce.org/2007/03/bytes.pdf"><img src="http://d.spearce.org/2007/03/bytes.png" height="216" width="360" /></a><br/><a href="http://d.spearce.org/2007/03/commits.pdf"><img src="http://d.spearce.org/2007/03/commits.png" height="216" width="360" /></a><br/><a name='more'></a><br/>The git.git repository (red line) appears to have a very stable history, but sees a huge spike right around now. This spike is an outlier caused by my development repository; I was very actively rebasing a branch on Feb 26th, resulting in hundreds of commits in my reflog.<br/><br/>The linux-2.6.git history seems to show a periodic cycle, with some days reaching up to 700 commits per day. The obvious correlation between number of KiB of disk space used and the number of commits per day is also clearly visible. As I am not a kernel developer the linux-2.6.git data came from a mirror of Linus' repository. I ommitted the first day outlier to prevent the Y-axis scales from shooting through the roof.<br/><br/>I have linked the images to higher-resolution PDFs, clicking on them will give you the PDFs.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com0tag:blogger.com,1999:blog-7214366369588984730.post-75891636365481453772007-01-21T18:57:00.000-08:002010-03-27T17:05:21.086-07:00git-gui Screenshotsniv on #git nudged me enough to create some screenshots of git-gui.<br/><a name='more'></a><br/><br/>The main window:<br/><image src="http://d.spearce.org/2007/01/gitgui-Main.png" /><br/><br/>Branch creation dialog:<br/><image src="http://d.spearce.org/2007/01/gitgui-CreateBranch.png" /><br/><br/>Delete branch dialog:<br/><image src="http://d.spearce.org/2007/01/gitgui-DeleteBranch.png" /><br/><br/>You can currently get git-gui by cloning the repository from repo.or.cz:<br/><pre>git clone git://repo.or.cz/git-gui.git</pre> or track it's progress through gitweb at <a href="http://repo.or.cz/w/git-gui.git">git-gui @ repo.or.cz</a>.Shawnhttp://www.blogger.com/profile/04158038862839573213noreply@blogger.com2