Work Around Address Book's Rollover Suite Bug to Calculate Age

This is a start to working around a known Rollover Suite bug in Apple "Snow Leopard" OSX 10.6.8's Address Book application, version 5.0.3 (as noted on Apple's support group). The application contains a really cool feature where one can write AppleScripts that appear as pop-up menu choices. A built-in example is the way you can click on an address field and "Copy Mailing Label" which does an okay job formatting a mailing label for that person's address.

This is accomplished by writing an Applescript that has the following form:

using terms from application "Address Book"
    on action property
        return "" -- a string of "maiden name", "phone", "email", "url", "birth date", "custom date", "related name", "aim", "icq", "jabber", "msn", "yahoo", or "address"
    end action property
    on should enable action for thePerson with theEntry
        return true -- if the menu choice should appear
    end should enable action
    on action title for thePerson with theEntry
        return "" -- the text of the menu choice to display
    end action title
    on perform action for thePerson with theEntry
        -- Perform the action if the user clicks the menu choice.
    end perform action
end using terms from

So if you plug in something like this:

using terms from application "Address Book"
    on action property
        return "phone"
    end action property
    on should enable action for thePerson with theEntry
        return true
    end should enable action
    on action title for thePerson with theEntry
        return "Dial number"
    end action title
    on perform action for thePerson with theEntry
        -- Dial the number.
    end perform action
end using terms from

And you save it as a compiled script in ~/Library/Address\ Book\ Plug-Ins/ (that is, your user account's Library/Address\ Book\ Plug-Ins/), then the next time you restart Address Book, it'll activate. If you click on the label for a phone number, you'll see a new menu choice like this:

Dial number Address Book menuAs I had mentioned, there's a known bug where this, in many cases, doesn't work—with birthdays, custom dates, or related names, for instance. I took a wild guess as to the problem and did make a workaround for birthdays and made a working way to simply display the age of someone in the "birthday" field.

The trick was that the Address Book application doesn't execute the should enable action, action title, nor (presumably despite not testing it explicitly) perform action script terms correctly. In short, it doesn't specify thePerson nor theEntry. For things like custom dates this is a problem because I think you'd need theEntry variable to figure out which custom date is being clicked. But for things like birthdays you can fake it by using the current selection of the address book.

At least assuming only one entry is selected.

So I made up a script that calculates the age of a person and adds that to a menu choice on jeir birthday. Here's the script:

using terms from application "Address Book"
    on action property
        return "birth date"
    end action property
    on action title
        tell application "Address Book"
            try
                set theCurrentPeople to selection
                if (number of items in theCurrentPeople) is 1 then
                    set theCurrentPerson to item 1 of theCurrentPeople
                    set theBirthdate to (theCurrentPerson's birth date as date)
                    tell me to set theAge to roundOff from (((current date) - theBirthdate) / (365.25 * days)) to 1.0E-3
                    return "Age " & (theAge as text) & "."
                else
                    return "Select one person."
                end if
            on error msg
                return "Error: " & msg
            end try
        end tell
    end action title
    on should enable action
        return true
    end should enable action
    on perform action for thePerson with theProperty
        return ""
    end perform action
end using terms from
on roundOff from num to precision
    set theFixed to round (num / precision)
    return (theFixed as number) * precision
end roundOff

And it looks like this:

Age in Address Book

 

117 total views, no views today


Removing Duplicate Messages in Apple's OSX Mail

I'm still running Snow Leopard (a.k.a. OS X 10.6.8) and using Apple's Mail version 4.6. I also tend to keep everything, so Mail is bloated with thousands of e-mails dating back to—believe it or not—October 14, 1990 (and checking, it's actually just shy of 100,000 messages). Over the years, Mail has created duplicate messages now and then. I did a bit of research and there are some tools to do it. For instance, Bohemian Boomer's article on Remove Duplicate Messages refers to the so-named AppleScript that does what it says. While I gather it would eventually work, scripting Mail with AppleScript is not exactly fast, and I found it was processing thousands of messages per hour and not the most robust architecture for such a huge project so I stopped it.

Nick Shubin's article on Finding Duplicates in the Mail Database talks about how each message is stored in the ~/Library/Mail/Mailboxes tree in its own *.emlx file. He used a program called Find Duplicate Files by Araxis which would probably work fine.

My technique was similar, although I didn't clean up the attachments files at all. I had written a Python script that would scan files and find duplicates. The original purpose was to replace the duplicates with UNIX "hard links"—where two or more files would point to the same data on disk (as opposed to a "soft link" where on directory entry points to another, much like a Finder alias). There's a whole history about how the data on disk is just stored in chains and is referenced by a link to the data's inode, so the UNIX remove (rm) command is also called the less-ominous and more-accurate unlink. (Something like that anyway … it's been decades since I looked at file systems that closely.)

Anyway, I added a couple options so I could delete files instead (omitting the ln commands to build the links). I closed Mail and ran it on my tree. In a half an hour, it found tens of thousands of duplicates. I had confidence in the program I wrote (having tested it in the past) so I went ahead and removed the duplicates.

Back in Mail, I figured it would be wise to select all the affected mailboxes and Mailbox:Rebuild them. Once I was done, I noticed a few duplicates still appeared in Mail.

It turned out—probably because of some odd rule I had added to Mail—that one message would have a color key in its XML extension and the other would not. Specifically, the difference between two duplicates was only:

<key>color</key>
<string>000000</string>

So I used TextWrangler to search all the files and simply remove that combination (figuring it was not harmful since it was setting some color to black which was probably the default anyway.) Running my script a second time, I got another few thousand messages and removed them. There may be more, but I haven't found them yet and I'm satisfied.

And yeah, someday I'll put that Python script in open source somewhere or other. For now, it's not adequately commented so I'll hold off on publishing it.

446 total views, no views today


Writing to Microchip's PIC Internal EEPROM Without Waiting

Background

I've been working on a project where I'd like to log statistical information about power-ons and run-time to the internal EEPROM on a Microchip PIC 12F683. I don't do much advanced programming so I'm using an old version of MPLAB (v8.88) with a PICSTART+ clone by Olimex and the CCS C Compiler version 4. I realize that combination of tools is not perfect, but it suits my needs.

I have had success using CCS's write_eeprom() functions. But first, let's go back a bit.

Reading the EEPROM data from a PIC is fairly straightforward: load the EEPROM address into the EEADR register(s) (just one byte to address the 256 bytes available on the little '683), set EECON1 bit 0 ("RD" bit) to 1, then read the data out of the EEDAT.

Writing, however, is a bit more complex. In the PIC 12F683 manual, there is a snippet of code to accomplish this1:

BANKSEL EECON1 ;
BSF EECON1,WREN ; Enable write
BCF INTCON,GIE ; Disable INTs
BTFSC INTCON,GIE ; See AN576
GOTO $-2 ;
MOVLW 55h ; Unlock write
MOVWF EECON2 ;
MOVLW AAh ;
MOVWF EECON2 ;
BSF EECON1,WR ; Start the write
BSF INTCON,GIE ; Enable INTS

The snipped highlights the part where 0x55 and 0xAA are loaded into EECON2 just prior to starting the write. This is the focus of most problems people have getting the EEPROM to write. But I was seeing something different.

The code that CCS C generates is basically the same:

MOVF INTCON,W
MOVWF intcon_temp
BCF INTCON.GIE
BSF STATUS.RP0
MOVLW 0x55
MOVWF EECON2
MOVLW 0xAA
MOVWF EECON2
BSF EECON1.WR
BTFSC EECON1.WR
GOTO $-1
BCF EECON1.WREN
BCF STATUS.RP0
MOVF intcon_temp,W
IORWF INTCON,F

It is not as careful about clearing the general interrupts (more on that in a minute), but it does wait for the write to complete before clearing the EECON1 write enable (WREN), and is clever about resetting the GIE by saving and restoring the whole INTCON register (although it's ultimately wasting a RAM location if you, as the programmer, know what state the GIE should be and can just BSF it if necessary). The big problem I have is that it takes around 5ms for the EEPROM to perform a write—and the write_eeprom() function blocks interrupts during that time. My application is to generate a precise-as-possible ~5KHz square wave using the timer interrupts, so the result of that delay is the clock "hangs" for 5ms each time a byte is written to the EEPROM. That's no good for me.

Now a little aside … I thought the Microchip method of clearing GIE was interesting. Their Application Note #576 outlines the reason for this:

To disable all interrupts, either the Global Interrupt Enable (GIE) bit must be cleared or all the individual interrupt enable bits must be cleared. An issue arises when an instruction clears the GIE bit and an interrupt occurs "simultaneously". For example, when a program executes the instruction BCF INTCON, GIE (at address PC), there is a possibility that an interrupt will occur during this instruction. If an interrupt occurs during this instruction, the program would complete execution of this instruction, and then immediately branch to the user’s interrupt service routine. This occurs because the GIE bit was not clear (disabled) when the interrupt occurred. Normally at the end of the interrupt service routine is the RETFIE instruction. This instruction causes the program to return to the instruction at PC + 1, but also sets the GIE bit (enabled). Therefore the GIE bit is not cleared as expected, and unintended program execution may occur.

They offer 4 workarounds in the application note. The one suggested in the data sheet is to keep clearing GIE and wait for it to stick.

The reason for all this hoopla is because the EEPROM module has some protection against runaway code writing over any good data in the EEPROM, and otherwise banging it to an early death:

  1. EEADR and EEDAT must be filled with the address and data to write.
  2. EECON1.WREN must be 1
  3. The sequence 0x55 then 0xAA must be written to EECON2, and then EECON1.WR must be set to 1 … umm … "quickly". I haven't looked very hard, but I haven't seen a specification for this.

If I were implementing the EEPROM module, I'd pick some small number of instruction clocks from the time 0x55 is written to EECON2 to the completion of the sequence of events. Written as pseudocode, something like:

Wait for 0x55 == EECON2.
Set count to 8.
If 0xAA != EECON2, decrement count; if 0 == count, goto start.
If 0 == EECON1.WR, decrement count; if 0 == count, goto start.
Start writing the EEPROM.

I do not believe there is any kind of program-memory reader in place that checks for the exact sequence of opcodes that forms:

MOVLW 0x55
MOVWF EECON2
MOVLW 0xAA
MOVWF EECON2
BSF EECON1.WR

Nonetheless, there's little reason to deviate from that—save for a compiler that might insert the BSF STATUS.RP0 to access the EECON* register bank after the first MOVLW 0x55. However, I've also read a thread that said that even an additional NOP would thwart the EEPROM write, so keep that segment of code tight.

Down The Wrong Rabbit Hole

I thought I'd get clever and reactivate GIE right after starting the EEPROM write, then let my main program loop twiddle its thumbs waiting for EECON1.WR to clear, all the while allowing the clock to run:

BSF EECON1.WREN
BCF INTCON.GIE
BTFSC INTCON.GIE
GOTO $-2
MOVLW 0x55
MOVWF EECON2
MOVLW 0xAA
MOVWF EECON2
BSF EECON1.WR
BSF INTCON.GIE
BTFSC EECON1.WR
GOTO $-1
BCF EECON1.WREN

But all of a sudden it's dead in the water: the EEPROM stays at its initialized values. Unfortunately, I don't have a way to see if the main loop is stopped as the interrupts keep running and the outputs clock like I expect.

It seems improbable that the insertion of BSF INTCON.GIE between setting EECON1.WR and waiting for it to clear would cause such a problem—especially since it's so similar to Microchip's own code (in that case, adding the code to wait for EECON1.WR to clear).

My debugging now turns to the interrupt handling. All the interrupts funnel into one interrupt handler which just looks at the timer interrupt. I took a closer look at the setup and this is how the CCS C sets things up:

BSF STATUS.RP0 ; enable_interrupts (INT_TIMER2)
BSF PIE1.TMR2IE
MOVLW C0 ; enable_interrupts (GLOBAL);
BCF STATUS.RP0
IORWF INTCON,F

What's interesting to me is that the compiler never explicitly sets PIE1 to anything, so PIE1.EEIE is not explicitly cleared which could be related to the cause of my trouble. Performing an inclusive-OR of 0xC0 = '1100 0000' to INTCON sets both GIE to allow interrupts, and PEIE to permit the peripheral interrupts to fire. Nonetheless, I fixed the code to clear all the peripheral interrupts in my startup code. That didn't work.

As a long-shot, I figured I'd try testing the EECON1.WR right away without doing anything about it. Perhaps there's a bug/quirk in the EEPROM write module that "needs" it to be read for the EEPROM write to proceed? Alas that didn't do it. So I figured I'd change my code to match the CCS compiler and just see if that did the trick: 5ms delays and all. Surprisingly I didn't observe the 5ms delays, but it did write the EEPROM … sometimes. And apparently I've got a bug somewhere that may have to do with byte ordering …

On the other hand, maybe EECON1.WR is cleared too fast after starting a write for some reason and I should be looking to PIR1.EEIF instead. This seemed to work better for me but I don't understand why. Could I have chips with a bug? That sounds extremely unlikely.

The way I got around the potential timing issues of disabling interrupts is to work around the threat in a different way. I'd set up the EEADR and EEDAT registers then set a flag to initiate the EEPROM write. The interrupt handler itself would issue the "magic sequence" and start the write while the main program loop would wait for the flag to clear then proceed with its own wait for the write to finish.

I made a debug block of code that, instead of trying to record legit statistics, just works its way through the EEPROM, loading each address with its address once a second. I let it run for the requisite 255 seconds and checked the results. There were no errors.

Read Errors?

So maybe I have the writing down … perhaps it's in the read? I thought about changing the code to add a delay between setting the EECON1.RD bit and fetching the data, so:

movf data,W
movwf EEADR
bsf EECON1.RD
nop
movf EEDAT,W

but I doubt that would have helped since my code wasn't working well. The first problem I was having—likely "doing wrong"—was to try and increment EEADR. I changed the code to increment a separate register then load EEADR with it didn't seem to work right either. The other potential problem was using a read-and-assign function which would take a reference to a long and then try and fill it in. I was hoping to use the function inline so I'd have access to the register as a constant but the compiler wouldn't let me—the idea was to movf EEDAT,W then movwf variable, and then likewise with variable+1, etc. The compiler, internally, could do this, but it insisted variable was a constant value. And unfortunately it decided to bizarrely use the indirect addressing functionality to add a bunch of code.

I reverted back to using the compiler's built-in read_eeprom() function and I finally met with success. The values I had were updating like I expected. I'll add a bit of code to verify each byte was written correctly (and rewrite indefinitely if necessary), but otherwise I'm confident things are working like I want.

The built-in read_eeprom() function has one additional quirk that my code didn't have: it clears bit 7 of the EECON1 register before setting EECON1.RD. If you're writing your own read routine, that may be worth checking out.

But a False Success

Unfortunately I still got very odd EEPROM behavior. I was finding that sort-of random values were being inserted in the EEPROM. See, I had 4 32-bit registers I kept logging. Seemingly at random, some of the registers were somehow set to register | 0x00000200 although the 0x02 value could appear at any byte, typically the second to lowest. And not always.

I kept working on the EEPROM write function, adding a bit of code that would re-read the value and keep trying to write it back. I also got generous with the write code, as apparently on the 12F683, messing with EECON1, EEADR, or EEDAT before both EECON1.WR cleared and PIR1.EEIF was set could cause write problems.

I also set up the write routine to reset EEADR to 0xFF (which I was not using) so any spurious writes would not affect any valid data.

But I was still getting the same problems. By now I figured the writing was correct and that there must be some error with the registers in RAM. I suspected the read routines so I decided to write my own.

They are particularly vanilla routines, merely reading a byte and putting it away (fetching 4 bytes in a row is left as an exercise to even the most inexperienced programmer). They only have two special features: before initiating a write, it waits for EECON1.WR to be cleared, and after the write is complete, it sets EEADR to 0xFF.

This has mostly cleared up my problems. I still see that spurious 0x02 appear, but it's much less frequently. I have no idea how to proceed from here.

Another Gotcha

Actually, several.

I found a Microchip forum topic that suggested to turn on brown-out reset, since a brown-out can cause random data to be written to EEPROM. It didn't affect my application at all since I was either using a solid 5V supply or the power was shut off and the 5V rail dropped to 0V.

Likewise, another forum topic suggests problems with power-supply decoupling capacitors could be at fault. A saggy rail when writing to the EEPROM could cause problems. Similarly, an inadequate pull-up resistor on the MCLR pin—if configured as a reset—could cause similar problems.

Power Loss Data Integrity

One other thing I realized was if the power was lost while the registers were being written, the code would never know there was a problem. Since I had the space, I made a second mirrored set of registers. Before starting to modify one set, the code sets a flag in the EEPROM by setting one byte to 0xFF. Once it finishes modifying that set, it resets the byte to 0x00. Then it repeats with the other set, setting a different byte to 0xFF, modifying the set, and resetting it to 0x00.

On boot-up, the code checks for the 0x00 in the right place. It uses the values from the first valid set it finds (under the assumption that writing completed successfully.) If both sets are invalid, it just resets all the values as they can't be guaranteed good.

I considered setting up a CRC check but decided it was not that critical. These logging values are not for any mission-critical function, so I didn't care if they got mysteriously garbled.

Conclusions

The PIC internal EEPROM has a number of quirks and requirements that are not immediately obvious:

  1. EEADR and EEDAT are not normal registers and shouldn't be treated as such. For instance, although valid, don't try and EEADR++ or incf EEADR.
  2. Reading data takes a couple instruction cycles, but writing data takes approximately 5ms that varies with temperature and input voltage.

For reading data from the EEPROM:

  1. Wait for EECON1.WR to be cleared in case a write is still in process.
  2. Set EEADR to the address to read.
  3. Set EECON1.RD.
  4. Read the value from EEDAT.
  5. If you are have trouble reading, try a BCF EECON1.7 before the BSF EECON1.RD (maybe only if you're using a PIC12F683). Also, try fetching the value from EEDAT as soon as possible after issuing the read request.
  6. Once done reading set EEADR to a location in EEPROM you are not using in case of spurious writes (particularly during a brownout/power loss.)

For writing data to the EEPROM:

  1. Set EEADR and EEDAT to the address and data.
  2. Wait for EECON1.WR to be 0 to ensure the last write finished.
  3. Set EECON1.WREN to 1 to allow the EEPROM to be written.
  4. Make sure interrupts can't delay execution of the "magic" sequence (either by executing this within a top-priority interrupt routine, or after disabling interrupts):

    MOVLW 0x55
    MOVWF EECON2
    MOVLW 0xAA
    MOVWF EECON2
    BSF EECON1.WR

  5. On some PICs like the 12F683, wait for both EECON1.WR to be cleared and for PIR1.EEIF to be set before changing EECON1, EEADR, or EEDAT.
  6. In theory you can re-enable interrupts and continue running code. But for better EEPROM protection, clear EECON1.WREN as soon as EECON1.WR is 0.
  7. Verify and retry all writes for added assurance.
  8. Once done writing set EEADR to a location in EEPROM you are not using in case of spurious writes (particularly during a brownout/power loss.)
  1. PIC12F683 Data Sheet, Microchip, 2007.

876 total views, 1 views today


Time Machine "The backup was not performed …" error

I just started getting a Time Machine error when backing up my Macintosh Mini (mid-2010) running "Snow Leopard" OSX 10.6.8. I get a charmingly cryptic error:

The backup was not performed because an error occurred while copying files to the backup disk.

The problem may be temporary. Try again later to back up. If the problem persists, use Disk Utility to repair your backup disk.

Well I tried repairing the backup drive to no avail. I'm now repairing disk permissions on my main hard drive. But what did help is a Dashboard Widget called Time Machine Buddy. If you run it as an administrator, you can view the backup log and see which file it's having a problem with. To be honest, I have no idea why it is suddenly taking issue with a number of files, but it is.

Upon further investigation, I'm getting an "Error -36" reading the file. This is not good—according to Apple's Common System Error Messages page:

This file is having difficulty while either reading from the drive or writing to the drive. The file may have been improperly written data to the drive or the hard drive or disk may be damaged. This is almost always indicative of a media error (hard error on the disk). Sometimes (rarely) it is transient.

Definitely not good … I guess I'll need to get a new main hard drive. I should be less surprised than I am: I bought the machine in September, 2010, so given how quickly things go bad, 5 years shouldn't be such a surprise. What a pain, though … I'm not looking forward to this.

Update 2015-Dec-3:

I noticed Time Machine started having issues with external drives as well which didn't jibe with a failing system drive. I found another suggestion to do a "full reset" on Time Machine. I shortened the procedure to:

  1. In System Preferences:Time Machine, click Options and remove any drives you can (certain drives are permanently added.) Then turn Time Machine "Off" and quit System Preferences.
  2. Eject the backup drive and power it off.
  3. Delete /Library/Preferences/com.apple.TimeMachine.plist. You'll need to have administrator privileges to do this.
  4. Mount the backup drive, reset any optional exclusions in the Time Machine preferences then start a backup.

I don't know if this will fix my problems—one thing I forgot to do was to repair the external disks (especially those causing problems.)

And another thing: if you don't want to use Time Machine Buddy, you can also review the system.log in the Console app. Again, you'll need administrator privileges to see the file, but you can type "backupd" in the search (actually "filter") box and review what Time Machine—a.k.a. backupd—is doing. I found this more useful than Time Machine Buddy for hardcore debugging—TMB is good to have if you've got an occasional problem and want to take a quick peek at what's up.

I also noticed an error concerning being unable to parse the SystemMigration.log. I found it in Console and it was from when I installed the system software 5 years ago so I deleted it. I'm also running a Disk Utility Repair on all the external drives.

337 total views, no views today


iCal Getting Slow When Showing a Different Day

So I've got a Macintosh Mini (mid-2010)—believe it or not, that's the official name—running "Snow Leopard" OSX 10.6.8. I synchronized a lot of data on that with a Galaxy Player 4.0 "YP-G1" (which I refer to as a "Palm Pilot" since it's shorter than saying "technically it's an MP3 player and not a smart phone", and since it replaced my Palm Vx which was in service for about 10 years.) I got frustrated with MissingSync for Android when it started flaking out and not connecting to the Palm Pilot anymore. I switched to SyncMate which I was very excited about (since, unlike MissingSync, it let me synchronize while charging.)

I started noticing that iCal was running slowly. Whenever I changed to a different day it would take around 4 seconds or so. I searched around and found iCal Dupe Deleter. It exports a backup for you in iCal then looks for duplicates and deletes them. Upon completion, you restore the fixed backup into iCal—a process which I assume replaces all your iCal data with the archive contents. I had 3 or 4 duplicates, but now iCal is (while not snappy) reasonably quick in switching between days. I only wonder if I simply created an archive in iCal with File > Export... > iCal Archive... then immediately restore it with File > Import... > Import... and picking the file I just created.

605 total views, 2 views today


Trouble with OsmAnd's "Smart" Merge of Favorites.gpx

Years ago I was quite excited by an application for my Samsung Galaxy Player Android called OsmAnd. It is a free application (although you can buy a non-free version to support the project) that allows you to download OpenStreetMap data and use it like a GPS. It supports routing and talking directions like a commercial GPS, but, given its OpenStreetMap roots, if you find an error, you can edit it yourself and within days the changes will percolate to everyone's device.

One of the things I grew fond of was to directly edit the favorites.gpx file which contained all your "Favorite" places (now called "My Places" as of version 1.8). I could find a location in, say, Google Maps, then take the latitude and longitude and create an entry in favorites.gpx. The same time I upgraded to version 1.8, I stumbled upon a file with some old locations I had saved on my (now dead) Garmin GPS. I did some text manipulations and dropped them into the favorites.gpx file, but they wouldn't import.

I played around with it for a while, and found that OsmAnd could open GPX files, with which it would try to import the entries. When I did that with favorites.gpx, it would read the file and ignore my new entries, replacing favorites.gpx with a version that did not contain any of the new entries. I didn't notice at the time, but it popped up a cryptic error message that thankfully led me to the problem:

Error reading GPX data Error parsing document. (position: line -1, column -1) caused by org.apache.harmony.xml.ExpatParser$ParseException: At line 4, column 16: not well-formed (invalid token)

The error reads "Error reading GPX data Error parsing document. (position: line -1, column -1) caused by org.apache.harmony.xml.ExpatParser$ParseException: At line 4, column 16: not well-formed (invalid token)". Examining that line in the favorites.gpx, I had attempted to include an ampersand (&) in one of the entries (the first one, as it turned out). Rather than coding it as the SGML entity (&amp;) I simply included it in the text which the parser (validly) didn't like. Unfortunately OsmAnd didn't handle the error very gracefully.

While I appreciate the new "smart merge" feature, I debate the use of the word "smart" in the way my poorly-formed gpx file was handled!

2,212 total views, 1 views today


About the FAX CSID/TSID on OSX 10.6.8 Snow Leopard

Call me a dinosaur. Here it is August 2014, and I am still using an Indigo iBook G3 running OSX 10.2.8 (Jaguar) on my home network because it can be used as a FAX machine. (And, well, for the approximately biannual Internet outage wherein I need to revert to dial-up and share the connection … which works surprisingly well for most day-to-day stuff. But I digress.) I've been running the long-discontinued PageSender software which was actually pretty nice, but the other day, it failed to bother to send a PDF. I figured it was a good idea to retire the iBook for its modem duties and just buy a USB modem. I opted for a USRobotics USR5637 56K USB Modem (from TigerDirect because I've been buying stuff from them since the 1990s).

It installed pretty easily and I was able to add it as a FAX modem per the System Preferences:Print & Fax page. One thing that I noticed was there was no place to enter the Called Subscriber ID (CSID) which is usually the same as the Transmitting Subscriber ID (TSID) and what I always thought was called the "Station ID" although my physical FAX machine's instructions simply refers to "Entering your name and phone number." In short, when you send a FAX, your phone number and TSID appear at the top of the page (and you can view the CSID of the FAX you just called). But in OSX 10.6.8 (Snow Leopard), there is only a space for "FAX Number".

My first reaction was to try and install PageSender to no avail: the installer failed to start on Snow Leopard.

So then I started digging. I started a free FAX account so I could experiment. I set the FAX number to my real FAX number and sent a FAX successfully. Both the FAX number and TSID appeared as that number. Next, I took a peek at the internal options: I opened a terminal session and (well, after some digging) entered:

defaults read /Library/Preferences/com.apple.print.FaxPrefs

Which resulted in:

{
EmailFax = 0;
FaxNumber =
(my FAX number);
PrintFax = 0;
PrinterID = "
(my default printer)";
ReceiveFaxID = "USRobotics_56K_Modem";
RingCount = 1;
SaveFax = 1;
SavePath = "
(my FAX receive folder)";
"device-uri" = "fax://dev/cu.usbmodem0000001";
}

It appears, therefore, that there is no place to enter a proper CSID/TSID. Tinkering with the preferences again, I tried entering text into the FAX number field—assuming it was indeed a substitute for the CSID/TSID. Indeed, this replaced "FaxNumber" in my FaxPrefs.

I looked around for modem logs, but that wasn't an option … only some error logs existed which could be accessed through the wackadoodle CUPS HTTP interface. If you have OSX, you can probably link to http://127.0.0.1:631/ and muck around with things you probably shouldn't touch.

Another thought was to try setting the CSID directly in the FAX modem. Once upon a time, Hayes-compatible modems connected through RS-232 interfaces and had internal non-volatile memory to store settings, although back then, the S-registers could only contain one byte and it was rare to find more than tens of bytes of storage inside a modem. I found an old article on Apple.com which defined a clunky way to do it and it got me to access the modem. At first I couldn't see what I was typing, so I entered an ATE1 to enable command echo. Man, that takes me back … the first time I typed it was somewhere around 1987 and the last time was probably more than 20 years ago.

There exists a "standard" Hayes-compatible AT command +FLID="local ID" which is used to set the local ID in a TSI or CSI frame. But it's only supported with modems that have Class 2 FAX support: AT+FCLASS=? returns 0,1,8 which apparently means the USRobotics modem supports data (0), Class 1 FAX (1), and voice commands (8). The latter of which interestingly implies I could use it as an answering machine … maybe.

Now that I'm invested a couple hours in, I thought I'd crack open the modem's internal documentation. Indeed, it has no support for the +FLID command which I already confirmed. Alas, the AT commands appear to be a dead-end as well.

In the end, I decided to leave the modem alone and just use my FAX number as a station ID. It is sufficient and effective—and it certainly wasn't worth the hours of fooling around … but once I get started on a short project like this, I like to see it through to the end.

1,859 total views, 1 views today


939 Days of Solar Power

So the number is a bit random, but there you go. I officially had my solar system connected to the grid on December 1, 2011 … 939 days ago. Based on my usage at the time, I was eligible for tax breaks and grants for a system that could produce about 4,140 kWh a year. The way the math worked out on my house, that meant a 4,140 watt system. I also had to get it grid-tied which means that the electric energy produced from the solar system was to be mingled into the electrical grid; that means I don't have any batteries, and if the power goes out, I don't have power either—even if the solar system had energy available.

Here's some things I learned.

A Few Basics About Electricity

Well, first a basic lesson in electricity. Current is a flow of electrons; voltage is a potential difference between two points of a number of electrons available. If there is a voltage (measured in volts), making an electrical connection through a device allows that potential difference to flow making current (measured in amps). The amount of power (in watts) can be calculated by multiplying the voltage by the current. That power can be used to do work: work or energy is power multiplied by time.

For instance, it takes work to pedal a bicycle one mile; an average person can continuously produce about 100 watts of power. If jee travels 10 miles per hour, it would take 1/10 hours to go a mile, so jee would have exerted 10 watt-hours of energy, or you could say jee did 10 watt-hours of work. Calories are also units of energy (although confusingly food-calories are Calories which are kilocalories or 1000 work calories). Nonetheless, it's about 9 kilocalories of work. (For a sanity check, a cycling calculator indicated you'd burn about 30 kilocalories, and since you'd burn about 8 kilocalories in that time on a 2,000 Calorie-a-day diet, that's 17 kilocalories which is at least in the ballpark.)

Solar Panel Basics

Sorry, I digress. Photovoltaic solar panels convert light to electricity. A panel is usually rated for its power in watts which is calculated under specific conditions—typically something like 1,000 watts per square meter which is the maximum energy from the sun. However, at any point, there is a specific insolation (the amount of sunlight reaching that point.) At the equator during an equinox, the sun is directly overhead, and one square meter sitting on the surface gets the full square meter of solar energy. But if you imagine tilting that panel: once you get to 90 degrees, the sunlight is hitting the edge, and none of it actually strikes the surface, so you'd get zero power, so at angles between, there is some percentage of sun hitting the panel.

On earth, two things are happening: the earth is rotating and the sun is perpendicular to the earth only at one latitude. So as a day goes on, energy from the sun starts hitting one spot on the earth at a very shallow angle, which slowly increases during the day until mid-day when it is as close to perpendicular as it gets (based on latitude) and then gradually decreases until sunset.

The point is if you have a solar panel and it's mounted to your roof, it will only produce its rated power if the sun hits it straight-on. So for a 100-watt solar panel, the naive expectation is that for 8 hours of sunlight, it would convert 800 watt-hours of energy in a day. But because of the latitude adjustment and because of the rotation of the earth, you won't get anywhere close to that number. For instance, my best day around the solstice last year was June 18, 2013 with 26.31 kWh produced over about 16.5 hours or an average of 1,600 watts—barely 38% of the system's rated capacity. And on the best day around the winter solstice (December 28, 2013), the system produced only 4.67 kWh over about 10 hours or 467 watts on average—only 11% of the system's capacity (that day, the peak output was only 1,870 watts or 45% capacity).

And then there's cloudy days which I'm just going to totally omit.

Well, enough about solar panels for now …

My System

The installed cost was $23,240 or $5.61/watt. That's probably about typical. The solar panels alone were $13,000 and the grid-tie inverter was $2,000. All the hardware and wiring cost another $3,000, so the solar stuff alone was $18,000. Permits were almost $2,000, and labor was $3,000, making up the rest.

Installation Grant

If you live in New York State like I do and you look at your electric bill, there is a charge buried in the fees that is called the "Renewable Portfolio Standard (RPS) charge" which is described as "a state-mandated charge that funds renewable energy projects to achieve targets established by the Public Service Commission." I believe this is what funds the state-level grant. At the time I installed my system, they were offering $1.75 per installed watt, up to a maximum of 40% of the system installed cost. That meant a grant of $7,245. One thing that might change is to offer an incentive for buy-back of excess power at a generous rate—also from that same fund, and replacing the installation grant. This is what Canada does—they offer to buy power at a rate of over $0.50/kWh; at that rate, my system would net about $2,000 a year.

Tax Breaks

Next, there were some serious tax breaks. On the federal side, there was a 30% credit on the installed cost, and New York State offered a credit of 25%. That meant about $4,800 on federal and $4,000 on state. Those tax credits are a little weird and I'm glad I have an accountant to handle it: since I didn't have that much tax to pay in any one year, he figured out how to apply it over multiple years. So going back to the dollars, the whole system was $23,240 minus $7,245 from the RPS funds, and I had to come up with $16,000. Over the course of several years, I got tax breaks totaling $8,800, so in the end, I had to pay $7,200 for the system.

Additional Costs

However, there was another cost that was kind of hidden. The solar system had to be connected to my breaker box, but I didn't have enough open slots for new breakers to hook it up. I had to pay about $2,000 to get a new electrical panel installed and get a new wire run from the pole to my house. I could have gone with a 150-amp service, but it was only a hundred dollars more to go with 200-amps and allow for a lot of future expansion (if I had to replace the 150-amp box, it would be another $2,000).

A New Electric Meter

I also got a new electrical meter with a digital display. But it's really confusing since it cycles between "001", "002", "F", and all-segments on (the last two are apparently for testing.) There's a blinking arrow as well which points left when the system produces more than is being used, and energy is going back to the grid or right for when the usage is higher than the production and energy is being drawn from the grid. When it's drawing from the grid, that's accumulated on "001", and when it's adding to the grid, that's accumulated on "002". In theory if I were producing exactly what I was using, neither value would increase. If you do the math, you can't determine how much solar energy has been produced. However, an additional meter in the basement shows exactly that (it's another digital, but it's nothing to do with the gas and electric company).

So on December 1, 2011, it all got switched on, and I started adding energy back to the grid. Well, you saw the numbers for December … nothing going back to the grid.

Producing and Consuming Energy Throughout the Year

The way it works is what I like to think of as an energy bank account. My utility company, RG&E (well, actually Spanish company Iberdrola, but that's another story) keeps track of the electricity usage and generation. On months when the system produces more than usage, the surplus energy gets added to the "bank account". On months when the system produces less than usage, energy is first removed from the "bank account" until none is left, and I start having to pay again.

You'll hear from installers that RG&E will "buy back" your excess. So I was naively thinking they'd buy it at the same rate they sell it which would be nice—something like $0.11/kWh in the end. Well, they buy back at the uselessly-named "avoided-cost rate" which is (theoretically) what they pay to buy from the national grid. That is more like $0.05/kWh. This past year they bought back 347 kWh at (exactly) $0.04956232/kWh for a whopping $17.20 credit on my bill.

Now, if you do nothing, they'll buy back the energy on the month you activated the system. So in my case, they would take the surplus I built up all summer and pay out at $0.05/kWh, and then for the rest of winter when the system wasn't producing enough, I'd be paying for electricity. However, you can call your energy supplier and change your "credit date" to something more useful. I took a guess and figured that April or May would be a good time so I set it to that. It seems to work out because, like I said, I had a surplus of 347 kWh and have not had to pay for any electricity.

Another gotcha with this system is that RG&E charges a fixed "customer charge" for the privilege of being hooked up to the grid. Last month it was $21.38, and that's been pretty steady.

Online Monitoring

On another note, my system is through SunPower, so there's an online monitoring system. I have access to more information (and after some time figured out how to get them to e-mail me summarized data each month) but you can see my system here. My installer said I'd love it, but it's all powered by Adobe Flash, so it's actually kind of annoying and difficult to use. If you find this helpful and want a system of your own, there's always this link that gets me a cash kickback if you get a solar system through SunPower.

Summary

Since it's been enough time, I can do the math on my rate of payback. Well, I can get pretty close anyway.

According to the data from SunPower, in 2013, the system generated 3,949 kWh of electricity (or, you might think of it as averaging out to a 450-watt power plant, perhaps in comparison to the Ginna Nuclear Generating Station that supplies most of the Rochester area—which by similar calculation is a 560,000,000-watt power plant). Based on data from RG&E, my home consumed 2,432 kWh from the grid and added 2,694 kWh to it. So supposedly of the 3,949 kWh the system produced, 2,694 kWh went back to the grid, leaving 1,255 kWh consumed in the house. Since I drew 2,432 kWh from the grid as well, that totals 3,687 kWh consumed. That leaves an excess of 262 kWh which is pretty close to the 347 kWh from my last bill.

Back in 2011 I was on ConEd's Green Power which was costing around $0.095/kWh with taxes and everything (but not counting the monthly charge which would have equated to $0.158/kWh). So just looking at the base cost of electricity, that's $583 in saved usage and $17 in extra production or an even $600 total. In the end, I paid $7,200 for the system, so assuming 2013 is an average year and electricity rates stay the same, it pays for itself in about 12 years. Of course, if the cost of electricity doubled, that means the system pays off in half the time. Nonetheless, having the capacity to generate electricity is a boon no matter what. And if you factor in the added value to the home, the system kind of pays off instantly.

But money isn't my motivation in this. I liked the idea of being part of a group getting us away from fossil fuel and nuclear usage. And if you figure the electricity I use is mostly produced just a few feet from where it's used. The U.S Government claims only 6% is lost from production to consumption, but I find that hard to believe as a typical high-power transformer is about 97% efficient, and at least four are needed from a generator to a household, so that's 89% efficient or an 11% loss. In any case, eliminating much of the grid offers possible gains in efficiency.

Addenda

June 27, 2014: Added the kickback link and wanted to mention that the ridge of my house runs almost due north-south, so my solar panels are actually installed on the west-facing side. While a better orientation, or a tracking system would use the panels more efficiently, there's also the factor of cost, and to be honest, it's not that big a gain to orient them differently.

August 9, 2014: Did a few grammar edits and added the bit about being a power plant.

Marcy 15, 2015: My payback period was way off so I recalculated the numbers and fixed it.

1,887 total views, no views today


ADC Adapter Board Shared at OSHPark

For those looking to just purchase an adapter board by the current design I've made, here is the link to the ADC Adapter Board at OSH Park. Good luck.

1,914 total views, 1 views today


Quirk about the ADC Connector

I started having problems with the ADC display using the adapter I built. It worked fine for quite some time then the display started to shut itself off and not be recognized by the system. Sometimes it would work to plug the cables back in.

At first I thought it was the power supply I was using. I don't have specs on it, but the problem surfaced when the temperature climbed. The computer area was hovering around 90°F, and I assumed the power supply was sporadically failing. I bought a new-to-me 24V 1.8A supply for my 17" monitor (I think that might be a bit on the low-side, but it was only a couple dollars at a thrift store.) The problem persisted almost immediately. So I figured the monitor might be fried.

The ADC connector shield is loose.

The ADC connector shield is loose.

I had noticed the shield on the ADC connector on the board was not connected to the ground very well. I didn't think much of it — after all, there were half a dozen ground wires already. But on a whim, I thought I'd add a couple dabs of solder and give it a solid connection to the pins soldered onto the board. Surprisingly, that did the trick. So far it's been 2 days and the monitor has not flaked out! It may be the power supply is on the low-side to begin with (24V) and without the additional ground of the shield, the voltage drop crept too high in the other wires for the monitor to function.

ADC connector shield.

ADC connector shield.

In any case, if you are observing sporadic problems where the monitor would shut off, apparently losing power, check the shield on the ADC connector and make sure it has a solid ground.

Solder added to ADC connector shield.

Solder added to ADC connector shield.

Update August 7, 2013: Not so fast. I am still having problems where the monitor will switch off and not come back on. Most recently, I had to unplug the power for 15 minutes as well as the connection to the computer before it came back to working. I'm not sure what it is, but the ground is not the only fault going on here …

6,891 total views, no views today