DealNews Posts

I worked for DealNews from 2006 to 2022. Over that time, I was privileged to work with some of the kindest, most talented people. This tag is mostly about day-to-day life and lessons slinging code.
MySQL
So I came across an interesting quirk in MySQL the other day. Let’s say you have a table schema and some values that look like this: +-------------------+------------------+------+-----+---------+-------+ | Field             | Type             | Null | Key | Default | Extra | +-------------------+------------------+------+-----+---------+-------+ | page_id       | varchar(30)      | YES  |     | NULL    |       | | clicks            | int(10) unsigned | YES  |     | NULL    |       | +-------------------+------------------+------+-----+---------+-------+ +---------+--------+ | page_id | clicks | +---------+--------+ | 1 | NULL | +---------+--------+ And then let’s say you pass the following SQL statement to MySQL: update page_click_count set clicks = clicks + 1 where page_id=1; If you come from a loosely-typed language such as PHP, you would probably expect clicks for page_id 1 to now be 1. But that’s not the case in MySQL. After the query is run, the table will still look like this: +---------+--------+ | page_id | clicks | +---------+--------+ | 1 | NULL | +---------+--------+ Not only does the query fail, but it fails with no warnings given. It appears that mathematical operations on null values silently fail. There are a couple of ways around this. The first and most obvious is to set NOT NULL and a default value on the column. In the example above, this would work. The NULL value in that field becomes a 0 and you can to normal mathematical operations on it. But what happens if, for whatever reason, you can’t do that? We actually have this situation in a few places at dealnews, where NULL represents a distinct value of that field that is different from 0. In this case, you can use COALESCE() to fill in the appropriate value for the field. update page_click_count set clicks = coalesce(clicks, 0) + 1 where page_id=1; Edit: Brian Moon informs me that this is actually part of the SQL specification. So hooray for specifications. Still, it’s kind of arcane; in working with MySQL (and PHP) for a decade now, this is the first time I’ve ever actually encountered this. Hopefully this helps someone who was as confused as I was.
Read More
Apache
The goal of this project were twofold: To completely eliminate the need for me to touch the phone to provision it. I want to be able to create a profile for it in the database, then simply plug the phone in and let it do the rest. And… To eliminate per-phone physical configuration files stored on the server. The configuration files should be generated on the fly when the phone requests them. So the flow of what happens is this: I create a profile for the phone in the database, then plug the phone in. Phone boots initially, receives server from DHCP option 66. Script on the server hands out the correct provisioning path for that model of phone. Reboots with new provisioning information. Phone boots with new provisioning information, begins downloading update SIP application and BootROM. Reboots. Phone boots again, connects to Asterisk. At this rate, provisioning a phone for a new employee is simply me entering the new extension and MAC address into an admin screen, and giving them the phone. It’s pretty neat. **Note: **there are some areas where this is intentionally vague, as I’ve tried to avoid revealing too much about our private corporate administrative structure. If something here doesn’t make sense or you’re curious, post a comment. I’ll answer as best I can. Creating the initial configs I used the standard download of firmware and configs from Polycom to seed a base directory. This directory, on my server, is /www/asterisk/prov/polycom_ipXXX, where XXX in the phone model. Right now we deploy the IP-330, IP-331 and IP-4000. While right now the IP-330 and IP-331 can use the same firmware and configs, since the IP-330 has been discontinued they will probably diverge sometime in the not too near future. With the base configs in place, this is where mod_rewrite comes into play. I added the following rewrite rules to the Apache configs: RewriteEngine on RewriteRule ^/000000000000\.cfg /index.php RewriteRule /prov/[^/]+/([^/]+)-phone\.cfg /provision.php?mac=$1 [L] RewriteRule /prov/polycom_[^/]+/[^/]+-directory\.xml /prov/polycom_directory.php` RewriteCond %{THE_REQUEST} ^PUT* RewriteRule /prov/[^/]+/([^/]+)\.log /prov/polycom_log.php?file=$1` To understand what these do, you will need to take apart the anatomy of a Polycom boot request. It requests the following files in this order: whichever bootrom.ld image it’s using, [mac-address].cfg if it exists or 000000000000.cfg otherwise, the sip.ld image, [mac-address]-phone.cfg, [mac-address]-web.cfg, and [mac-address]-directory.xml. So, we’re going to rewrite some of these requests to our scripts instead. Generating configs on the fly We’re going to skip the first rewrite rule (we’ll talk about that one in a little bit since it has to do with plug-in auto provisioning). The one we’re concerned with is the next one, which rewrites [mac-address]-phone.cfg requests to our provisioning script. So each request to that file is actually rewritten to provision.php?mac=[mac-address]. Now, in the database, we’re keeping track of what kind of phone it is (an IP-330, IP-331 or IP-4000), so when a request hits the script, we look up in the database what kind of phone we’re dealing with based on the MAC address, and use the variables from the database to fill in a template file containing exactly what that phone needs to configure itself. For example, the base template file for the IP-330 looks something like this: <sip> <userinfo> <server <?php foreach($phone as $key => $p) { ?> voIpProt.server.<?php echo $key+1 ?>.address="<?php echo $p["host"] ?>" voIpProt.server.<?php echo $key+1 ?>.expires="3600" voIpProt.server.<?php echo $key+1 ?>.transport="UDPOnly" <?php } ?> /> <reg <?php foreach($phone as $key => $p) { ?> reg.<?php echo $key+1 ?>.displayName="<?php echo $p["first_name"] ?> <?php echo $p["last_name"] ?>" reg.<?php echo $key+1 ?>.address="<?php echo $p["name"] ?>" reg.<?php echo $key+1 ?>.type="private" reg.<?php echo $key+1 ?>.auth.password="<?php echo $p["secret"] ?>" reg.<?php echo $key+1 ?>.auth.userId="<?php echo $p["name"] ?>" reg.<?php echo $key+1 ?>.label="<?php echo $p["first_name"] ?> <?php echo $p["last_name"] ?>" reg.<?php echo $key+1 ?>.server.1.register="1" reg.<?php echo $key+1 ?>.server.1.address="<?php echo $p["host"] ?>" reg.<?php echo $key+1 ?>.server.1.port="5060" reg.<?php echo $key+1 ?>.server.1.expires="3600" reg.<?php echo $key+1 ?>.server.1.transport="UDPOnly" <?php } ?> /> </userinfo> <tcpIpApp> <sntp tcpIpApp.sntp.address="pool.ntp.org" tcpIpApp.sntp.gmtOffset="<?php echo $tz ?>" /> </tcpIpApp> </sip> The script outputs this when the phone requests it. Voila. Magic configuration from the database. There’s a little bit more to it than this. A lot of the settings custom to the company and shared among the various phones are in a master dealnews.cfg file, and included with each phone (it was added to the 000000000000.cfg file). Now, on to the next rule. Generating the company directory Polycom phones support directories. There’s a way to get this to work with LDAP, but I haven’t tackled that yet. So, for now, we generate those dynamically as well when the phone requests any of its *-directory.xml files. This one’s pretty easy since 1) we don’t allow the endpoints to customize their directories (yet), and 2) because every phone has the same directory. So all of those requests go to a script that outputs the XML structure for the directory: <directory> <item_list> <?php if(!empty($extensions)) { foreach($extensions as $key => $ext) { ?> <item> <fn><?php echo $ext["first_name"]?></fn> <ln><?php echo $ext["last_name"]?></ln> <ct><?php echo $ext["mailbox"]?></ct> </item> <?php } ?> <? } ?> </item_list> </directory> We do this for both the 000000000000-directory.xml and the [mac-address]-directory.xml file because one is requested at initial boot (the 000000000000-directory.xml file is intended to be a “seed” directory), whereas subsequent requests are for the MAC address specific file. Getting the log files Polycoms log, and occasionally the logs are useful for debug purposes. The phones, by default, will try to upload these logs (using PUT requests if you’re provisioning via HTTP like we are). But having the phone fill up a directory full of logs is ungainly. Wouldn’t it be better to parse that into the database, where it can be easily queried? And because the log files have standardized names ([mac-address]-boot/app/flash.log), we know what phone they came from.Well, that’s what the last two rewrite lines do. We rewrite those PUT requests to a PHP script and parse the data off stdin, adding it to the database. A little warning about this. Even at low settings Polycom phones are chatty with their logs. You may want to have some kind of cleaning script to remove log entries over X days old. Passing the initial config via DHCP At this point, we have a working magic configuration. Phones, once configured, fetch dynamically-generated configuration files that are guaranteed to be as up-to-date as possible. Their directories are generated out of the same database, and log files are added back to the same database. It all works well! … except that it still requires me to touch the phone. I’m still required to punch into the keypad the provisioning directory to get it going. That sucks. But there’s a way around that too! By default, Polycom phones out of the box look for a provisioning server on DHCP option 66. If they don’t find this, they will proceed to boot the default profile thats ships with the phone. It’s worth noting that, if you don’t pass it in the form of a fully-qualified URL, it will default to TFTP. But you can pass any format you can add to the phone. if substring(hardware, 1, 3) = 00:04:f2 { option tftp-server-name "http://server.com"; } In this case, what we’ve done is look for a MAC address in Polycom’s space (00:04:f2) and pass it option 66 with our boot server. But, we’re passing the same thing no matter what kind of phone it is! How can we tell them apart, especially since, at this point, we don’t know the MAC address. The first rewrite rule handles part of this for us. When the phone receives the server from option 66 and requests 000000000000.cfg from the root directory, we instead forward it on to our index.php file, which handles the initial configuration. Our script looks at the HTTP_USER_AGENT, which tells us what kind of phone we’re dealing with (they’ll contain strings such as “SPIP_330”, “SPIP_331” or “SSIP_4000”). Using that, we selectively give it an initial configuration that tells it the RIGHT place to look. <?php ob_start(); if(stristr($_SERVER['HTTP_USER_AGENT'], "SPIP_330")) { include "devices/polycom_ip330_initial.php"; } if(stristr($_SERVER['HTTP_USER_AGENT'], "SPIP_331")) { include "devices/polycom_ip331_initial.php"; } if(stristr($_SERVER['HTTP_USER_AGENT'], "SSIP_4000")) { include "devices/polycom_ip4000_initial.php"; } $contents = ob_get_contents(); ob_end_clean(); echo $contents; ?> These files all contain a variation of my previous auto-provisioning configuration config, which tells it the proper directory to look in for phone-specific configuration. Now, all you do is plug the phone in, and everything else just happens. A phone admin’s dream. Keeping things up to date By default, the phones won’t check to see if there’s new config or updated firmware until you tell them to. But his also means that some things, especially directory changes, won’t get picked up with any regularity. A quick change to the configs makes it possible to schedule the phones to look for changes at a certain time: <provisioning prov.polling.enabled="1" prov.polling.mode="abs" prov.polling.period="86400" prov.polling.time="01:00" /> This causes the phones to look for new configs at 1AM each morning and do whatever they have to with them. Conclusions The reason all this is possible is because Polycom’s files are 1) easily manipulatable XML, as opposed to the binary configurations used by other manufacturers, and 2) distributed, so that you only need to actually send what you need set, and the phone can get the rest from the defaults. In practice this all works very well, and cut the time it used to take me to configure a phone from 5-10 minutes to about 30 seconds. Basically, as long as it takes me to get the phone off the shelf and punch the MAC address into the admin GUI I wrote. I don’t even need to take it out of the box!
Read More
Asterisk
At dealnews, as I’ve written before, we run Asterisk as our telephone system. I find it to be a pretty good solution to our telecom needs: we have multiple offices and several home-based users. And, for the most part, for hard telephones, we use Polycoms. We run mostly IP-330s, with a couple of IP-4000s and a few new IP-331s. We also have softphones, a couple of PAP2s and a couple of old Grandstreams from our original Asterisk deployment in 2007 that I’m desperately trying to get out of circulation. But it’s mostly Polycoms. Recently, I changed how we were doing provisioning. I’ll write a more in-depth post about this later, but the short of it is that since Polycom phones use XML for their configuration information, we now generate them dynamically instead of creating a configuration file. It’s what I should have done back in 2007 when we bought our first round of Polycoms. But this presented me with a problem: how do I re-provision the older phones - some of which I don’t have easy physical access to (at least that doesn’t involve an airplane ride) - to use the new configuration system? In doing some research, I discovered that Polycom allows you to set, via certain commands, the provisioning server from within a config. With this information, I crafted a custom re-provisioning config that looks like this: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <deviceSettings> <device device.set="1" device.dhcp.bootSrvUseOpt.set="1" device.dhcp.bootSrvUseOpt="2" device.net.cdpEnabled.set="1" device.net.cdpEnabled="0" device.prov.serverType.set="1" device.prov.serverType="2" device.prov.serverName.set="1" device.prov.serverName="server"/> </deviceSettings> And included it at the top of the 000000000000.cfg file (one of the default files downloaded by each Polycom phone): <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <APPLICATION APP_FILE_PATH="sip.ld" CONFIG_FILES="update.cfg, phone1.cfg, sip.cfg" MISC_FILES="" LOG_FILE_DIRECTORY="" OVERRIDES_DIRECTORY="" CONTACTS_DIRECTORY=""/> Then, using Asterisk, I issue the check-config command: asterisk*CLI> sip notify polycom-check-cfg peer The phone should reboot, pick up its new config, then reboot again with with proper new provisioning information from the new provisioning provider. Next post, I’ll show you how to use PHP and mod_rewrite to eliminate the need for per-phone config files.
Read More
Apple
Every day, when I get to work, there are a number of tasks I do. Among the first thing I do is connect to a number of servers via SSH. These servers - our development testing, staging, and code rolling servers - are part of the development infrastructure at dealnews. So every morning, I launch iTerm, make three sessions and log into the various servers. Over time, I’ve written some helper scripts to make this faster. My “go” script contains the SSH commands (using keys) to log into these machines so that all I have to do is type “go rpeck” to log into my development machine. Still, this morning, the lunacy of every morning having to open iTerm and execute three commands, every day without fail, struck me. Why not script this so that, when my laptop is plugged into the network at work, it automatically launches iTerm and logs me into the relevant services? Fortunately, iTerm exposes a pretty complete set of AppleScript commands, so with a little work, I was able to come up with this: tell application "System Events" set appWasRunning to exists (processes where name is "iTerm") tell application "iTerm" activate if not appWasRunning then terminate the first session of the first terminal end if set myterm to (make new terminal) tell myterm set dev_session to (make new session at the end of sessions) tell dev_session exec command "/Volumes/iDisk/bin/go rpeck" end tell set staging_session to (make new session at the end of sessions) tell staging_session exec command "/Volumes/iDisk/bin/go staging2" end tell set nfs_session to (make new session at the end of sessions) tell nfs_session exec command "/Volumes/iDisk/bin/go nfs" end tell select dev_session end tell end tell end tell What this little script does is, when launched, checks to see if an instance of iTerm is already running. If it is, it just creates a new window, otherwise creates the first window, then connects to the relevant services using my “go” script (which is synchronized across all my Macs by MobileMe). Then, with it saved, I wrap it in a shell script: #!/bin/bash /usr/bin/osascript /Users/peckrob/Scripts/launch-iterm.scpt And launch it with MarcoPolo using my “Work” rule that is executed when my computer arrives at Work. Works great!
Read More
DD-WRT
To celebrate the re-launch of my “blog,” I’m going to do a multi-part entry about DD-WRT. But, first, a little history. For the first time in 10 years, I have no servers running in my house. At one point, I had three servers running in here doing various things. Then, I moved my public server offsite (it’s in the rack at the office now). That left two more Gentoo boxes running here in the house. Late last year I picked up a 1TB external hard drive, which I attached to my iMac and deactivated the file server. I will probably eventually replace this with a Drobo FS, but for now it’s fine. That just left a single Gentoo box that was running Asterisk and various network services. But I finally convinced my wife to let me drop the goofy VoIP line that I was paying $30 for and just add more minutes to her cellphone. With Asterisk out of the picture, the only thing left running on that box was network services. Well, a few weeks ago I ordered a TP-Link TL-WR1043ND router, intending to use it as a testbed for DD-WRT. Well, my experiments worked so well that I pulled my old router out and replaced it with the DD-WRT one. The faster processor also afforded a nice speed bump of about 7 Mb/s. With it handling all the services, I pulled out the final server and deactivated it. And my office is blissfully quiet now. DD-WRT is now handling all the minor network services (DHCP, NTP, etc). But what is it about DD-WRT that makes it so awesome - awesome enough to rip out some of my network infrastructure to make way for it? A few things that I will cover in this post. 1. DHCP static address assignments Believe it or not, the built-in firmware of the WRT-54G did not give you the ability to define a static address to be assigned by DHCP based on MAC address. This seems like a glaring oversight to me, but it was the reason I ran my own DHCP server rather than use the built-in ones. In DD-WRT (v24-sp2) you can go to the Services tab and set as many as you’d like. In my case, these are a couple of devices (like printers) that are addressed via IP address by the various machines, as well as my laptop and iMac. So that’s one nice thing, but it’s not nearly as cool as … 2. VPN Support The standard and VPN versions of DD-WRT support both PPTP and OpenVPN varieties of VPN … and I’m actually using both at the same time. My router is both a VPN server and VPN client as well. How? Why? Well, as to why, at dealnews, we run a PPTP-based VPN to allow us to work at home as needed. Once connected, we have access to our testing servers and all our development services. It’s like being directly connected to the work network, but I’m sitting at my iMac at home in my pajamas. I had been connecting directly from my Macs to the VPN for some time but, sitting at home the other day, I reflected on how silly it was that I was connecting two machines to the VPN and only when I needed them, rather than using DD-WRT to have a single tunnel up all the time that any computer on the home network could use if needed. Setting up a PPTP VPN Endpoint using DD-WRT So how did I set it up? Trial and error, as, frankly, the DD-WRT documentation is a bit lacking. So if you find yourself in my position of wanting to have a tunnel to your workplace VPN, hopefully this documentation will help you. I’m making a few assumptions before we begin: You have already configured your router using DD-WRT and have the most recent release (as of this writing, v24-sp2), VPN version installed. The version number should be in the upper right corner of the web admin. If it says “std” or “vpn,” you’re in good shape. If it says “micro,” you probably don’t have the necessary tools. You possess some basic understanding of networking, and have the necessary settings to complete a VPN connection. If you’ve gotten as far as flashing with third-party firmware, you probably do. You understand that there is the possibility, albeit remote, that you could brick your router. I am not responsible for that, which is why I suggest you purchase an additional router to get all this set up on first before sacrificing your primary router. With that out of the way, let’s begin! Log into your router’s DD-WRT web admin, and go to the Services -> VPN tab. Under PPTPD Client, click the radio button next to Enable. In the “Server IP or DNS Name” box, enter your VPN server. In the “Remote Subnet” box, enter the network address of the remote network. In my case, this was 10.1.2.0. In the “ Remote Subnet Mask” box, enter the remote subnet mask. In my case, this was 255.255.255.0. In the “MPPE Encryption” box, I have “mppe required,no40,no56,stateless”. This was required to get mine to work, but may not be necessary for you. Try first without it, then try with it if it won’t work. Leave the MTU and MRU values alone unless you know what you’re doing. Enable NAT. Username and password are self explanatory. WIth that done, press “Save” and “Apply Settings” at the bottom the page. With any luck, you should now have a VPN tunnel up to your remote host. To test it, go to Administration -> Commands, and in the command box, enter the following: ping -c 1 <some remote address on VPN> If you get a response back that looks like: PING <remote service IP> (<remote service IP>): 56 data bytes 64 bytes from <remote service IP>: seq=0 ttl=64 time=281.288 ms --- <remote service IP> ping statistics --- 1 packets transmitted, 1 packets received, 0% packet loss round-trip min/avg/max = 281.288/281.288/281.288 ms Then it’s up and working. Now, try from your computer… Probably didn’t work, did it? This is because your router’s firewall doesn’t yet know about the remote network or to route packets to it appropriately. For some reason, the current version of DD-WRT does not add the appropriate configuration to the firewall automatically when the PPTP tunnel is established. So, we have to do it manually. Go to Administration -> Commands, and enter the following: iptables -I OUTPUT 1 --source 0.0.0.0/0.0.0.0 --destination <remote network address>/16 --jump ACCEPT --out-interface ppp0 iptables -I INPUT 1 --source <remote network address>/16 --destination 0.0.0.0/0.0.0.0 --jump ACCEPT --in-interface ppp0 iptables -I FORWARD 1 --source 0.0.0.0/0.0.0.0 --destination <remote network address>/16 --jump ACCEPT --out-interface ppp0 iptables -I FORWARD 1 --source <remote network address>/16 --destination 0.0.0.0/0.0.0.0 --jump ACCEPT iptables --table nat --append POSTROUTING --out-interface ppp0 --jump MASQUERADE iptables --append FORWARD --protocol tcp --tcp-flags SYN,RST SYN --jump TCPMSS --clamp-mss-to-pmtu At the bottom, press “Run Commands” and wait. It shouldn’t take long, and should produce no output. Then, enter that command again, and press “Save Firewall” at the bottom. Give your router a few seconds to restart the appropriate services, then try again from your computer. Your machine, and all machines on your network, should now be able to access the VPN. In this configuration, only traffic matching the remote network will pass over the VPN - the rest of your traffic will be routed to the Internet in normal fashion. Now, in my next entry, I’ll tell you why I’m not using PPTP to connect myself back to my home network when I’m on the road.
Read More
Apple
At dealnews, we have an internal Jabber server that we use for our internal communications. As part of that, we have a number of internal chat rooms for the various areas of the company. I’m a big believer in automation - that is, scripting various repetitive actions that I have to do every so often. One of these little things is joining our developer chat channel each morning when I get to the office. Unfortunately, there’s no built in way in Adium to do this, nor does Adium expose native AppleScript commands to join group chat. It does for other functions, but group chat functionality is conspiciously absent, even though there’s a long standing feature request to implement this. So, we have to hack it. In this case, I used AppleScript to imitate keyboard input set CR to ASCII character of 13 tell application "System Events" tell application "Adium" to activate keystroke "j" using {command down, shift down} keystroke "development" keystroke CR end tell So we have a script, but how to automate the launching of it? I mentioned MarcoPolo before. It has quickly become one of my favorite pieces of Mac software. In this case, I use MarcoPolo to launch the AppleScript (with a 10 second delay to allow time for Adium to start and connect to the Jabber service). You can launch AppleScripts using the osastart utility like so: /usr/bin/osastart /Users/codelemur/Scripts/DevChat_AutoJoin.scpt It sucks that it’s like this, and I wish they would expose a more native way to do this, but it does work.
Read More
Apple
I know top X lists are almost passe at this point, but that’s not going to stop me from giving a shout-out to some of the applications that daily make my life easier: MarcoPolo MarcoPolo is a neat little application that is capable of executing actions based on a set of rules. That is, if something on the system changes (such as an IP address, power status, USB or even the light level), it can execute a series of commands (such as mounting network drives, setting the screensaver, changing the default printer, etc). It can even run arbitrary shell scripts! Why this is useful to me: At dealnews, we (the dev team) all use MacBook Pros for our development work and constantly alternate between home and office. Whenever I arrive at work in the morning, the minute I plug my MacBook into the network, MarcoPolo senses that the IP address has changed from my home and changes the default printer, mounts some network shares, adjusts the screensaver settings, and runs a few other custom shell scripts I have to set up my environment. All without having to do a single thing. When I get home, it executes still more commands to change to a remote development environment. Completely effortless. XMeeting XMeeting is a SIP softphone (and videoconferencing application, but I’ve never used the video features) that allows you to connect to a SIP server and place calls using your laptop. Why this is useful to me: At dealnews, we run Asterisk as our phone system (see my earlier posts on Asterisk). One of the many nice features of Asterisk is its standards compatibility - that is, you can use anything that can talk SIP with Asterisk. Since CounterPath has apparently decided that Leopard compatibility for their free softphone (X-Lite) is not a priority, XMeeting comes to the rescue. As a bonus, it actually acts like a Mac application and doesn’t do the stupid things that X-Lite did (like messing with the system volume). Quicksilver Quicksilver is the single application I cannot live without. On a Mac without it I am almost lost. More than just a launcher, it is a tool to help you work more efficiently. You can press Ctrl+Space and type what you want and Quicksilver will launch what you need. That’s a horrible description for how cool this app is. **Why this is useful to me: **Without Quicksilver, I am lost. It makes it literally so fast to move around your Mac without taking you hands off the keyboard. A quick hit of Ctrl+Space gives you the ability to launch programs, open files, navigate contacts and send emails, and make quick notes among many othe things that this program can do. It is essential to my everyday life as a Mac user. DejaMenu DejaMenu is a neat little program that will display the current application’s main menu as a popup menu where the mouse is whenever a key combination is pressed. **Why this is useful to me: **I use my MacBook Pro with a second monitor when I’m at the office. One of the things that has infuriated me for awhile as a Mac user with multiple monitors is the inability to have the top menu bar either on each monitor respresenting the application on that monitor, or the ability to have it move with whatever monitor the mouse is on. It’s irritating to have to go back to the main monitor when the application is running on a different one. DejaMenu allows you to pop the application menu wherever your mouse is, which makes things a little easier. Additionally, I mapped the key combination to a button on my Logitech MX-1000 to make things even easier.
Read More
Python
So it’s been awhile since I’ve written. In that time, my girlfriend has moved in here with me in Huntsville and, as always, dealnews has kept me very busy. However, it has not prevented me from occasionally trying my hand at something new. A week or so ago I decided that I was going to learn Python. However, as part of my nature, I simply can’t “learn” a language without having a purpose. For instance, I have never been able to simply read a book on programming - I needed a reason. So I’ve been giving myself reasons to do little tasks here and there in Python. One of them came to me just today. I have recently moved all of my development at dealnews from the PC to a Macbook. I’ve never been an OS-bigot - always use the right tool for the job, and the Mac - which in many ways is just Unix with pretty make-up - is the perfect platform. However, I still use many of the peripherals I purchased for my PC, including my Microsoft Natural Egronomic Keyboard that I adore. At home, I still use a PC (until I can afford a new Mac Pro), albeit with the same keyboard. One of the things I really love about the keyboard is that it has various buttons that are just … buttons. They can be mapped to do anything you want them to. There are five multi-function buttons at the top that can be mapped to run programs. So I’m sitting here thinking, “self” (because that is what I call myself), “why not write a little program to run on the click of that button and go to the next or previous track in iTunes, so that changing the music doesn’t involve any more effort out of my busy programming day than hitting an additional keystroke”. But, it must work both at home and at work, meaning that it must run in Windows and Mac. Enter Python I knew from previous experimenting in .NET that iTunes exposes a COM object on Windows. With that in mind, I quickly found this page that described almost exactly what I wanted to do in Windows. So that left the Macintosh. After an hour or so of digging on Apple’s website, I found this page that described how to access the COM on the Mac - and wouldn’t you know, the functions are slightly different. After that, it was pretty easy: import sys from optparse import OptionParser platform = sys.platform if platform == "win32": import win32com.client iTunes = win32com.client.gencache.EnsureDispatch("iTunes.Application") if platform == "darwin": from Foundation import * from ScriptingBridge import * iTunes = SBApplication.applicationWithBundleIdentifier_("com.apple.iTunes") def previousTrack(): if platform == "win32": iTunes.PreviousTrack() if platform == "darwin": iTunes.previousTrack() def nextTrack(): if platform == "win32": iTunes.NextTrack() if platform == "darwin": iTunes.nextTrack() def main(): parser = OptionParser() parser.add_option("-n", "--next-track", action="store_true", dest="next") parser.add_option("-p", "--prev-track", action="store_true", dest="prev") (options, args) = parser.parse_args() if options.next == True: nextTrack() if options.prev == True: previousTrack() if __name__ == "__main__": main() So yeah. It’s kind of code monkeyed together, but not bad for someone who’s only been doing Python for a week in the evenings. Passing either a -n or -p to the script causes it to command iTunes to go forward or back. Of note, to work on Windows, it does need the COM components from the Python for Windows extensions. I’m gonna expand this script some more in the future, but for now it does what I need.
Read More
Asterisk
Hello there! I figure that if I’m going to start using this blog to post the wanderings and wonderings of a mid-level engineer at a dot-com company (I work at dealnews to be specific, and I guess I should include the standard disclosure that my employer does not endose or support anything that I say/do here), perhaps I should give some substance to my first post. So, I figure I would write a post on something I have plenty of experience with: PHP. But what to write about? Surely, there must be ten million PHP tutorials on the ‘net and I don’t need to add to the noise already out there as to what are/aren’t the best practices using PHP, so I thought about using PHP in some lesser known areas. And here is one lesser known, but very cool area: you can use PHP to route phone calls! At a previous employer, I worked with Asterisk as a software development consultant. My primary role was to build web interfaces to Asterisk (and other telecom hardware) backends, though while working as a consultant I learned quite a bit about extending Asterisk to do crazy cool things. “It’s Just Software!” Asterisk is an open-source software PBX that was created by Mark Spencer (an Auburn grad and now CEO at digium). It is quickly becoming a challenger in the PBX market (fact: we use it at dealnews), and an entire industry has sprung up around Asterisk and open-source IP telephony. For the purposes of this tutorial, I’m going to assume that you already have Asterisk installed and configured to your liking, and are now wishing to extend it beyond what it is capable of doing with the builtin dialplan applications. If this is not a good assumption in your case, may I highly suggest the Asterisk Tutorial at voip-info.org, or even better, the O’Reilly Asterisk book, which is a little dated but still quite relevant to most beginner-level stuff. Meet AGI, CGI’s hard-working cousin: AGI, or the Asterisk Gateway Interface, is the key to extending Asterisk beyond what it is capable of doing on its own. AGI gives Asterisk the ability to run and interact with scripts and programs outside of Asterisk. AGIs can be written in any language that can be executed on a Linux system (and there have been AGIs written in PHP, Python, Perl, C, Bash and just about every other language out there). Since PHP is my language of choice, that is what I’m going to concentrate on in this tutorial. Asterisk AGIs are actually incredibly simple creatures. When run from within the Asterisk dialplan, they simply send commands to Asterisk using standard output and read the results on standard input. Its what happens between those that is really, really cool. Enough Talk! Code or GTFO! So, let’s get started! First, you need to set up your script environment. I recommend doing this in an include-able file so that you can reuse it in future AGIs. There are a few commands you need to know about: <?php // This turns on implicit flushing, meaning PHP will flush the buffer after // every output call. This is necessary to make sure that AGI scripts get their // instructions to Asterisk as soon as possible, rather than buffering until // script termination. ob_implicit_flush(true); // This sets the maximum execution time for the AGI script. I usually like to // keep this set low (6 seconds), because the script should complete pretty // quickly and the last thing we want it to do is hang a call because the script // is churning. set_time_limit(6); //This sets a custom error handler function. We'll get back to this later. set_error_handler("error"); //This creates a standard in that can be used by our script. $in = fopen("php://stdin","r"); //This creates an access to standard error, for debugging. $stdlog = fopen("php://stderr", "w"); ?> Okay, that’s not too bad! Now, we’re going to do a little more advanced stuff. Every time an AGI script executes, Asterisk passes a number (about 20) values to the script. These AGI headers take the form of “key: value”, one per line separated with a line feed (\n), concluding with a blank line. Before we can do this, we need to write a few functions to read from AGI input, write to Asterisk, Execute commands, and write to the Asterisk CLI. These are the functions I use: <?php function read() { global $in, $debug, $stdlog; $input = str_replace("\n", "", fgets($in, 4096)); if ($debug){ fputs($stdlog, "read: $input\n"); } return $input; } ?> So what are we doing here? Well, the first line, we strip out the line feed in each chunk we get from stdin. Then, we check to see if $debug is set and, if so, echo what we read to standard error. Finally, we return the line we just read. Pretty simple, right? Well, this little funtion will save you lots of time. Next, we need a way to write data: <?php function write($line) { global $debug, $stdlog; if ($debug) { fputs($stdlog, "write: $line\n"); } echo $line."\n"; } ?> This function is even more simple: it just writes out to standard error if $debug is on, and outputs whatever was sent to it with an additional new line. This next function, however, is more complex. <?php function execute($command) { global $in, $out, $debug, $stdlog; write($command); $data = fgets($in, 4096); if (preg_match("/^([0-9]{1,3}) (.*)/", $data, $matches)) { if (preg_match('/^result=([0-9a-zA-Z]*)( ?\((.*)\))?$/', $matches[2], $match)) { $arr['code'] = $matches[1]; $arr['result'] = $match[1]; if (isset($match[3]) && $match[3]) { $arr['data'] = $match[3]; } if($debug) { fputs($stdlog, "CODE: " . $arr['code'] . " \n"); fputs($stdlog, "result: " . $arr['result'] . " \n"); fputs($stdlog, "result: " . $arr['data'] . " \n"); fflush($stdlog); } return $arr; } else return 0; } else return -1; } ?> Woah, complex! Well, not really. execute() is the swiss army knife of AGI programming: it allows you to do interactive stuff inside this AGI script. First, as you can see, it calls the write() function we just wrote, writing an AGI command to Asterisk. Then it looks for a response on standard in. A response from Asterisk takes the form of “result=<result> <data>”. So, we use preg_match to get this out for us and put it into something usable. We do the debug output again, then return the array or 0 or -1 in the event of failures. Just two more functions to go: <?php function verbose($str,$level=0) { $str=addslashes($str); execute("VERBOSE \"$str\" $level"); } function error($errno,$errst,$errfile,$errline) { verbose("AGI ERROR: $errfile, on line $errline: $errst"); } ?> As you can see, these two functions are very simple. One gives verbose output to the Asterisk CLI, and the other is the error function we declared using set_error_handler above. Back to reading in variables. Now that we have the ability to read in, let’s read in the default variables that are passed to the script by Asterisk. We do this using the following code chunk: <?php while ($env=read()) { $s = split(": ",$env); $key = str_replace("agi_","",$s[0]); $value = trim($s[1]); $_AGI[$key] = $value; if($debug) { verbose("Registered AGI variable $key as $value."); } if (($env == "") || ($env == "\n")) { break; } } ?> This creates an $_AGI associative array (in the spirit of $_POST, $_GET, etc) for you to use containing all the items Asterisk passed in. For each read() line, in the first line we split it to get the key and value (this could probably have been done better with a regular expression, but I got a copy of some AGI code from a friend and modified it many moons ago before I began using regular expressions). Then, we strip out the “agi_” that Asterisk adds to the key because it is superfluous, and trim out the spaces and other garbage from the value, adding them to an array. Putting It All Together: Congratuations! You now have all the tools necessary to write an AGI! I suggest (as above) putting those in an include so you can reuse as necessary. So what next? Now, you write an AGI script! Let’s start with a simple example: #!/usr/bin/php <?php include "agi.php"; execute("SAY DATETIME #"); ?> That simple! Of course, all this AGI does is read the date and time to the caller, then exit, but it just shows that AGIs can do really powerful things, really simply. “Calling” you AGI: So now you have this AGI written and you want to use it, but you don’t know how. Well this is pretty easy too! AGIs should be placed in whatever directory you define for “astagidir” in your asterisk.conf file. Unless you changed it, this will be /var/lib/asterisk/agi-bin. Next, be sure that the file is executed by setting the executable bit “chmod +x ". You may also have to fiddle with the permissions: the asterisk user or group need the ability to read and execute the script. Then, you just call it from your dialplan, like so: exten => 1000,1,AGI(<filename>)` Now, after you “extensions reload” of course, you should be able to dial 1000, and watch your AGI spring into action! A more complex example: This is an AGI I wrote at dealnews when someone in the office requested the ability to custom set names to caller IDs and have it work on all phones. Keep in mind that this is only half of the solution (the other half is a web interface). #!/usr/bin/php <?php include "agi.php" ; $db=mysql_connect('redacted', 'redacted', 'redacted'); if(!$db) {     verbose("Could not connect to DB!"); } if(!mysql_select_db('redacted', $db)) {     verbose("Could not use DB!"); } $res=mysql_query(sprintf("select substitution_name from cid_substitution where from_number='%s'",$_AGI['callerid'])); $result=mysql_fetch_row($res); if(sizeof($result)) {     execute(sprintf("SET CALLERID \"%s <%s>\"",$result[0],$_AGI['callerid'])); } mysql_close($db); ?> This demonstrates one of the main advantages to using AGIs, and PHP in particular: the ability to easily interact with databases. In this program, I’m using the caller ID supplied by the carrier to fetch a corresponding name from a database and send it back along with the call. Routing calls is accomplished by calling the EXEC function with DIAL, giving you the ability, with a little work, to route calls based on the database. Pretty neat for a language thought of only as web coding. Indeed, there is a large list of commands that AGIs can use, and variables passed into them, available here. Help! It doesn’t work! Relax! Problems happen from time to time. One of the most common faults is forgetting to set the +x bit on the file to make it executable. Permissions problems are also relatively common. For More Information: voip-info.org - a.k.a. “the wiki,” is the major information repository for Asterisk knowledge specifically, and IP telephony in general.
Read More