Apps, Data Analytics, Projects, Oh My
It has been a while since my last update, and no, I haven’t abandoned Let’s Fish.
In short the project ran into the least exciting bottleneck imaginable being an enormous amount of data.
Waterbodies, rivers, flowlines, and drainage areas sound fairly manageable until the computer starts counting them in the tens of millions. Then every “small” design decision becomes hours of processing, a frightening amount of disk space, and a new opportunity for the database to make me scream.
What really slowed everything down wasn’t writing the code. Most of that moves pretty quickly. It was processing the national datasets, verifying the results, recovering storage, and making sure a stopped or crashed job could continue without starting over or quietly corrupting something.
That last part has already paid off. One of the long database runs recently crashed, recovered, and resumed from the last safe checkpoint. It was annoying, but it also proved the system could take punches without forgetting several weeks of work. That’s progress.
The foundation is much farther along now. The major national water layers have been prepared and verified, and the current work is turning millions of raw waterbody records into stable identities the rest of the platform can build on.
The vision is still the same. Help answer where I should fish, what I might catch, what the current conditions look like, what’s legal, how I can get there, and why one destination makes more sense than another.
I’ve also been working on a separate model-validation project, a value investing trading platform, handling some boring-but-important business and planning work, keeping this site moving, and occasionally remembering that I’m supposed to have a life outside databases. All the projects are interconnected so they’re happily skipping down the yellow brick road to the Emerald City.
I’m keeping the next stages vague for now. There is still plenty to build, and I’ve learned not to celebrate a finish line while the computer is actively moving it.
Let’s Fish is still moving forward. It’s just currently doing so at the speed of millions of tiny database transactions.
Apparently the fish are patient. Sometimes I’m not.
Leave a Reply