Netbook project Initiated in Dec 2003(?) by Mike Lloyd at Xios (how met?) 2.4.19 kernel port had been done by Solution Foundry in Canada - a long-time contractor. Working with montavista on filesystem. But had had compoilation problems and generally not terribly satisfied. Xios said we would put in a bid based on Debian. Peter Naulls put together an image based on debian arm on a CF card with XFCE4, dillo, xterm, which was 400Mb but impressed them greatly. Stuff just worked, it was responsive, and had taken about 2 weeks to do. This was a hack done by taking a standard debian-arm installation, throwing out as much as possible (e.g docs) and bodging in an x-server for the epson display chip. Psion were interested so we wrote a proposal for apps that would do what they wanted. This was accepted on 1st April, so suddenly we had stolen the contract from under montavista - quite a coup. All we had to do now was deliver a lot of somewhat untested technology in about half the time it really needed. One thing that is important on any job is to get people who know what they are doing involved. Mike talked to the Debian-arm nexus that conveniently exists in Cambridge, and assemebled an initial team of me, Peter Naulls, and Phil Blundell. Project split into kernel work and filesystem work. We were the filesystem team. Psion a tricky company to work with due to being technical enough to know what is going on and want to stick their oar in, but not necessarily good at making choices, especially when things get sticky, and in the unfamiliar area of free software. They were keen on the whole Linux thing - having realised what was possible, but were very unfamiliar with an open development process, and we had to explain to them how things worked. They kept wanting to make copies of things (in cvs) and then change them slightly, rather than just make a small patch which made the necessary change, and could be easily moved forward to newer versions. Defining the project. Next we had to stick a finger in the air and work out what needed doing and how long it would take. and write a spec. beta rc1,2 SDK ??? Mike - I think there were specific phases in contract terms - one the for the alpha peter did, one for writing the spec (was that just the 4 of us?) then the last part for doing the whole thing, or maybe that was split into beta and final? ???? Can you give me ballpark figures for the contract sizes and anything you could write about the process of quoting would be useful. ???? Can you remind me how the kernel stuff went. At what point was 2.6 recommended and when did we suggest using simtec? (I recall chocky started a port, and was rude about SF then vince took over, but I forget the timing). External developers. Having sold Psion on the Debian concept and the potential of external developers Psion set up a meeting (march?) of a mumber of interested people - the core team plus a number of Debian developers (Kinnison, the DPL, Kyllikki, and quite a few psion people). They handed out Netbooks with the Debian imgae ona dn signed everybody up to NDAs, then told us what a cool project this was and we discussed various technical things about apps to use, ways of making it fit etc. (we'll come back to this later) With the spec taking shape we started having a lot of meetings. Psion is a proper corporation and it _loves_ its meetings. We were all traipsing down to london every week for a few weeks until Mike managed to curtail it and point out that developers need to be developing. So he did a lot of meeting-attending on our behalf and we were very grateful. The project had a deadline of 1st Oct for final ROM image so thing could be shiped in october in time for Xmas and trade shows. This seemed a bit odd as the device was not aimed as the consumer Xmas market, but it had been decided on high and was not negotiable. Once we stood back and looked at what needed doing we realised we had a real job on our hands to get everything that was necessary done in the 6 months available. (mike - when did we start exactly?)?? So now the team consisted of: Raf (psion) - head honcho Liza (psion) - Marketing, Chris Jakob (psion) - Technical head Brian Iddon - production manager Andy Mclernan - QA and server admin NIck Healey (slashdesign) - UI designer Liam Girdwood (Wolfson) - X Costas stylianou (psion) - hardware guy/X Alex Loveday (psion) - Abiword Apps team: Mike (Xios) - head honcho Peter N (himself) - openoffice + various Phil Blundell (Nexus) - OEguru/GPE/Desktop/X/mozilla Wookey (Aleph One) - external developer liason, filesystems, printing Florian Boor (kernel Concepts) - GPE Robot101 (aleph One) - OE helper Olga Kubica - tester Paul Webb (aleph One) - docs Aided by: Charles Briscoe-smith - server admin Billy Villa Shonner (Xios) - graphics kernel team: Ron statler (SFI) - SFI bossman Mark Whitaker (SFI) - 2.4 porter Vince sanders - 2.6 porter Later: Vince Wolfe - project manaer (later) Russel King - guru and fixer for 2.6 Philippe de swert - help External devs: Martin Sevior - abiword Jody GOldberg - gnumeric Pawel Salek - balsa Mozilla guy Totemguy USB guy As you can see that's a lot of people. So communication was important. We had a server running a wiki, mailing list, bugzilla (and a web forum everyone ignored). We also set up an IRC channel on freenode. This wasthe first time I had really used IRC, and I was very impressed with how effective it is for efficient communication with a team in a hurry. Only the geeks used this which was good because it allowed us to be rude about the management if necessary :-) So the project spec was hastily drawn up. Psion had a pretty good idea what they wanted it to do - provide typical 'PC' functionality but with instant-on, in a long-battery-life, lightweight package. So necesary apps were WP, Spreadsheet, email, browser, PIM stuff (calendar, notes, todo), PDF reader, networking. Preferably it should do IM, printing, games, text editor, database too, but there wereobviously going to be severe filesystem size contraints (64Mb flash) and we weren't sure at thisstage how much could be crammed in. We had already identified a load of apps that would do the job but had to make some choices - which email client (sylpheed, balsa, kmail), which browser (firefox, konqueror, minimo), and so on. Choices were madeonfunctionaility and footprint size. After looking at our app lists it was easy to exclude anything kdelibs based and decide to stick with gnome/gtk based stuff so as to keep dependencies sensible. This mean we had to lose the relatively svelte konqueror and go with firefox or minim - neither of which was slim! Started with sylpheed - changed to balsa as it needed fewer libs for html support, and author was keen to help. Psion really liked XFCE4 that had been in the demo, but we persuaded them that matchbox wasa much better fit - ther screen being 800x600 is very small by modern standards so having overlapping windowing is just a waste of space. All previous psion palmtops have had the same concept - switchable windows, not overlapping windows. We picked GPE to provide the desktop infrastructure and PIM apps. It was small and PB was familiar with it. Other apps were gnumeric, gpdf, gaim, totem, modem dialler, linneighborhood, image viewer, cups-manager, MS compatibility very important. Idiot-proof interface important (get rid of everything). There was a huge amount of work to put all this together with a nice desktop, consistent icons, strip down apps, fix bugs, make everything work better with a touschscreen interface in 6 months flat. Psion made it very clear that there was a tearing hurry on. nevertheless we were confident it was perfectly do-able - the variables being how many niggles were left at the end. Of course we had to guess at extaly how many hours it would all take which was a bit of a finger in the air job. We all guessed at our relevant areas. Mike doubled it, and quoted Psion 100K for the whole thing. Staged payments on deliverables (what were they exactly Mike??) Round about now we actually made the first significant mistake but it was not clear until much later. We were working like typical free-software project - make an approximate spec to start with, knowing that some of it will have to be changed later as we find issues. But Psion was the sort of big company that will keep pointing to the original spec saying 'this is not in - your candidate has failed acceptance testing', even when schedules are starting to slip and everyone is working like crazy to deliver something that works. It turned out that we should have written the spec much more explicitly about what would and would not work. That would have taken an extra 2 or 3 weeks and still it would not have been complete, but it could have saved us a lot of money in the end. Bear this in mind if you end up doing something like this again. The company may not take the sensible flexible approach that a free software project does. A secondary deliverable was openoffice. This clearly wouldn't fit in the 64Mb :-) butwould go on a 512Mb CF along with evolution. that would be the 'pro' version for the power user needed better MS doc compatibiility. Psion agreed the quote/spec and we set down to serious work. Build environment: Considered scratchbox and emdebian but plumped for OE, due to it's relative maturity, good GPE integration and PB's familiarity. Set up nightly build machine, and CVS. Really useful to be sure everything works, but tricky in terms of deliverables (see later). Initially tracked OE for as along as possible before we had to fork and picka version and stick with it for the rest of the project. OE moving along at a great rate. First big job was to get apps into OE. This turned out to be a big deal and took bloody ages. A lot of dependencies. every library and app need to be oeified so it would cross-build in the oe environment. This is often not simple. weeks passed. We thoug 'beta' meant 'mostly working but a bit ropey'. Psion thougt it meant finished to all intents and purposes except a few bugs. It was at this point that we realised things might get a bit sticky. The daily builds meant that every day there was always a better version available so it was difficult to get psion to take one and test it for acceptance. But we needed the money... Eventually had to have moratorium on checkins to force psion to take a version. We all did our own dev for a week then checked in a pile of stuff. primitive but effective. USeful bit of psych management there. Lots of technical problems: -------------------------- qulaity of free software not good. missing icons causes segfaults, balsa invisible window, gnuemric can;t read any numbers. Lots of GTK probs with touchscreen. App inconsistencies. balsa crashed on receiving HTML mail. turned out to be GDKpixbuf problem in GTKHTML library. Normal Xservers more forgiving than Kdrive, Busybox 1.00-rc1 and rc2 were horrible broken (insmod). This came at a very bad time for us. Lots of plain development on GPE apps for keybaord and larger screen use, and on desktop. JFS2 on NAND quite new. on > 32B NAND never been tried before. wired FS problems, due to paired 32K 'clusters' and nandwrite and mkjffs2 options. Keyboards turned outto be tricky. Psion has funcky keymap and wierd overloadings. Nasty interaction between kernel and X. Xmodmap. shift-caps-lock breaks if you let go of shift beforecaps-lock. 'help' key delivers samekeycode as another key due to way 'mediumraw' keycodes are delivered - escaped wiht preceeding null if more than 7bits. X server doesn't understand this. Practical difficulties moving OE build environment fromtest machine at Aleph1 (debian unstable, with X) to Psion server (debian stable, no X). revealed bugs in OE, like some of gnome using pixmap conversion tool (gdk-pixbuf-csource) in libgtk-2.0 on host instead of in OE host environment. Again happened when there was a great deal of pressure to get it up and running. Exchange server at psion wasold andcrappy version that did IMAP badly, particularly partial transfers. Itsimply didn't deliver the bytes it said it would. This made itlook like 'balsa doesn't work' to psion people, althogh it worked everywhere else. more general issues ------------------ Kernel dev running in parallel. Affected keyboard and flash use and MMC (no driver for that for a long time). 2.4 driver not unmountable. Big argument about kernel versions and responsibilities. 2.4 team still plugging away after everone else had stopped using it. Fixes going into wrong versions. Not sure what lessons to learn from that bit Mike - and comments??? Very hard to ge through to mamagement that sticking with old kernels is not smart, even though it feels 'safer'. Very hung up on 'changing the version number' (from 2.4.19 to 2.4.26) even though this fixes USB and loads of other stuff. New project manager part way through - just what we didn't need.Changed the desktop round and wanted all new icons. Starting to get a bit dilbert now... Tester was invaluable ofr actually getting bugs categorised, triaged and dished out. Bug-handling process worked well, although we put off using it for as long as possible in order to get free-flowing dev done without complaints about 'regressions' all the time. Having periods when heavy breakage was expected was very useful. Psion kept looking back at the alpha and saying 'but this used to work',not really appreciateing that by moving from debian to OE we had changed to a whole new codebase with differeent problems. Psion were impressed with geek hours (checkins at 3am showing that we were doing our utmost :-). Things now slipping noticeably, and it started to become clear that the budget wasn;t going to cover the dev time. Negotiations required with subcontractors to accept fixed-price deal so we all get to chare the pain. However we reckoned that assuming it was good then wewould go on to 'phase II' as which point we would have abetter idea of future costs and could all getback some money for the extra hours we were now all putting in. SO it was worth persevering and anyway we had all got quite involved in the project and wanted to make it work for our own satisfaction. real struggle getting money out of Psion for ocmpleted segments - they have a culture of non-paymentin finacne dept. THis is particularly problematic for small companies. On the one hadn they desperately want stuffdone now, on the the other they won;t pay for it. External developers - very effective. Abiword guy fixed 5 killer bugs in 10 hours work that Psion developer hadn't been able to find in 6 weeks work. Strongly recommend getting these peope involved if needed. Abiword guy very excited at sshing into wired amr machine in UK from wireless link in australia to run gdb on tricky problem :-) Linuxworld- device well-received. People quite impressed. Much effort - things really coming together - even help system when bonbshell dropped. Psion weren't going to continue with project after it was delivered. That rather changed things. We now had a good idea of how difficult they could be about minor acceptance issues and there were several hundred bugs in bugzilla. We could imagine them strwtching out acceptance forever and us having to do another couple of mohths work effectively for free' on a dead project, in order to get paid. Effectively we went on strike for afew days to back mike up in negotiating a fairly hard limit on the duration that theyu could string out the bugs for before taking a version and declaring it OK or not. ??? what else should I mention? The aftermath from Xios POV? What would you o different next time? Later: The external developers thing was very odd. They got everyone excited with the initial meeting and set up a mailing list for them but then never once released an image to them to play with or asked for any help. Some more hands (who weren;t demanding cash) would have been useful as things developed, but Psion seemed to have forgotten all about them. How long did things really take? Looking through the mail pile: Proposal written 27th March, Accepted 31st 1 month: setting up server, forum, lists, wiki, training Psion people on debian, how to cross-compile, patch getting more people in team. setting up build system Initial tech analysis - which filesystems, how laid out from 1st May Start hacking: GNumeric could load a spreadsheet continaing just 1,2,3,4. LibGSF suffered from arm float syndrome (4-bytes arranged little-endian, 2 words arranged big-endian). sparc: 40 09 21 fb 54 44 2d 18 (big endian) x86: 18 2d 44 54 fb 21 09 40 (little endian) arm: fb 21 09 40 18 2d 44 54 (mixed endian) first OE build. 'new alpha' June spec completed 9th june end may couple of weeks in june spent making 'demo' version based on the debian alpha work for psion to wave at people - fettling old 2.4 kernel and mostegregious bugs, and getting somepaower management. None of this was progress on the 2.6+OE based 'actual' version. start arranging external developers. Allow a week or two for this. decide minimo is not ready and go for firebird/fox PB alpha 21st June (OE GPE build+hacked in debian packages) July Vince Wolfe arrives at Psion UI specs written based on alpha - discussed with devs and changes started faster machine for OE lots of work oe-ifying major apps Balsa bug - (crashes on receiving HTML mail - GDK bixbuf problem) OPenoffice ported to ARM (world first) Psion not very happy by end month due to things like balsa bug (regression from debian version), and no dailys yet. Aug: Tester arrives (olga) a week of 16hr days...in order to get: Automated daily builds on nexus (finaly get everyone to stop using hacked-up alphas) 5th abiword in OE 8th took OE snapshot and forked. 10th beta candidate - move to proper QA with bugzilla, testing all checkins 13th manuals work starts balsa exchnage debugging. balsa UI changes (mailbox count, new wizard, speedups, infoprints,GPE address-book interfacing, new menus gnumeric UI changes (new menus) abi UI changes (new menus, reduced toolbars) Sept: hardware to external developers lots of bug-fixing, e.g. abiword not keeping up with caret, gnumeric scrolling too far when key held Beta failed accetpance on a few points. 26th philippe joined to do help end sept RMK brought in on kernel Oct: more balsa UI changes nand support in 2.6 -> flash filesystem option Getting CF cards,MMC card to auto-appear and disappear (udev (MMC)/hotplug/pcmcia (CF/PCMCIA) subsystems, and filemanager) 12th remember there is suppposed to be a database! 20th pre-release Integrating Helpsystem (uses GPEhelpviewer) end oct - told that project is dead. Nov: 2nd confirmation that rest of contract will be paid if we deliver RC, they test it for a week and we fix those fatals bugs and regressions.. Then on final RC2 build we get the rest of the dosh. This should happen by Nov 22. Tech features: Had to have a ramdisk to load modules. Initially because psion pcon chip module was non-free. Later because it made it easy to have scriptage to boot of internal nand or CF according to presence of valid FS. Modules lived in ramdisk - struggled to get all needed stuff in kernel+ramdisk image (limited to 4Mb by bootloader) by the end. Having a build system that checked boot image size was a really good idea so we knew when it would break on nightly build. busybox, ash, throw out libm etc. Touchscreen: kernel needed nobbling not to produce 0presure events all the time, needed to get rid of glitches, tricky to get balance between corners on fast lines and ? calibration util integrated into X, lot of work needed on GTK (so menus work). hold-gets-right-click proved problematic. You have to give a left click first. Gnumeric specifically prevents a right click whilst dragging. If a left button up happens then other apps give things you didn't want. and individual apps: make icons bigger, make draggable areas wider, change logic in some apps (gnumeric prevents . make double-clicks allowable further apart, and slower). Added infoprints - very handy, especially for anything slow. Nand layout 4 Mb for kernel+ramdisk 56Mb for system 4Mb for user data automounting. udev scripting for MMC/USB (magic device numbers) pcmcia scripting for CF/PCMCIA mount with -s so things can be yanked. This actually worked very well (eventually) Needed an 'eject' button to foce unmount for a while, but eventually that became unnecessary Hardware: 64Mb nand flash SA xscale pxa255 @ 400Mhz 800x600 screen 256Mb RAM CFslot MMC/SD slot PCMCIA slot pcon power management chip Epson video driver chip Wolfson audio chip Cypress USB chip Outcomes: OO for arm, Lots of major apps in OE (gnumeric, libgsf, linneighborhood, cups-manager, cups, cupslib, firefox, balsa, abiword, xmonobut, detect-stylus, libfribidi, aspell, OO - 20MB binaries, 800Mb source. 3-day build on fastest arm machine available (iyonix), takes 4Gb. Cross-compiling it is _really_ hard.