My blog passed 50,000 page views today. I want to thank everyone who has visited and helped me pass this goal.
Sunday, March 27, 2016
Saturday, March 26, 2016
Low Power Scheduling Library for Arduino 328U
I wrote my own scheduling library for Arduino a couple of years ago. One of the to-do's for that project was to replace the main loop with a sleep/power down feature that would reduce power down but still accurately wake up and perform the tasks when required.
The reasons that you want to use less power for a microcontroller project is easy. You want your project to run longer on a set of batteries. You want your solar powered project to work with the smaller, cheaper solar panel. You want your coin cell powered camera trigger to work for years, instead of months.
Last weekend I sat down to actually program the power saving feature that I had imaged a couple of years ago. My first few dozen tries failed, and on Sunday of last weekend all I had working was to put the processor in idle mode. Which I took as a win, because at least the sleep functions of my program were working.
I read a lot of various web pages about how to use the watch dog timer on Arduino, but the one that actually let me figure it out was the post that starts with "OK, I think I have it" on this web page. I used that example program as a simple example that worked to figure out what I was doing wrong. I found that example this weekend, after poking at things all week long and going to sleep thinking about the problem.
The first draft of my code that actually worked was just uploaded to github. There were just 2 tricky changes I had to make to my library code.
The first one was the fact that the watchdog timer can't be set to trigger at any arbitrary length of time. It only triggers at discrete units of time, sort of like a quantum energy emission. So I put a loop in that found the largest block of time that could fit in the scheduled time, and went to sleep for that amount of time, then it would wake up when the watch dog timer fired and find the next largest remaining block of time that could fit in the delay we needed. If there is a little bit of time left that is less than the 15ms minimum time that is left, I just delay.
The second thing was the fact that my code requires the use of the millis() function call, but that the timer in the micro-processor that keeps track of time is suspended when the chip is in power down mode. So to fix this I just drag all the scheduled tasks ahead by the amount of time that I just slept. A side effect of this is that because this timer almost never runs, I think I fixed the 50 day timer wrap problem. :D
One of the most difficult things to figure out was that for some reason the function to set the watch dog timer delay was not working, you had to set the register directly. From the example code that helped me figure this out.
For my own code I had to set the same register with this line:
And because Science requires documentation, I used my multi-meter to measure the difference between my old and new scheduling libraries using my clock sketch.
As you can see, the change is nearly a 50% power savings. And this is not even optimizing the hardware being used to reduce wasted current. How would you like your Arduino project to run for 18 hours instead of 10 hours on one charge?
Improvements.
The reasons that you want to use less power for a microcontroller project is easy. You want your project to run longer on a set of batteries. You want your solar powered project to work with the smaller, cheaper solar panel. You want your coin cell powered camera trigger to work for years, instead of months.
Last weekend I sat down to actually program the power saving feature that I had imaged a couple of years ago. My first few dozen tries failed, and on Sunday of last weekend all I had working was to put the processor in idle mode. Which I took as a win, because at least the sleep functions of my program were working.
I read a lot of various web pages about how to use the watch dog timer on Arduino, but the one that actually let me figure it out was the post that starts with "OK, I think I have it" on this web page. I used that example program as a simple example that worked to figure out what I was doing wrong. I found that example this weekend, after poking at things all week long and going to sleep thinking about the problem.
The first draft of my code that actually worked was just uploaded to github. There were just 2 tricky changes I had to make to my library code.
The first one was the fact that the watchdog timer can't be set to trigger at any arbitrary length of time. It only triggers at discrete units of time, sort of like a quantum energy emission. So I put a loop in that found the largest block of time that could fit in the scheduled time, and went to sleep for that amount of time, then it would wake up when the watch dog timer fired and find the next largest remaining block of time that could fit in the delay we needed. If there is a little bit of time left that is less than the 15ms minimum time that is left, I just delay.
The second thing was the fact that my code requires the use of the millis() function call, but that the timer in the micro-processor that keeps track of time is suspended when the chip is in power down mode. So to fix this I just drag all the scheduled tasks ahead by the amount of time that I just slept. A side effect of this is that because this timer almost never runs, I think I fixed the 50 day timer wrap problem. :D
One of the most difficult things to figure out was that for some reason the function to set the watch dog timer delay was not working, you had to set the register directly. From the example code that helped me figure this out.
void myWatchdogEnable() {
// turn on watchdog timer;
// interrupt mode every 2.0s cli();
MCUSR = 0;
WDTCSR |= B00011000;
WDTCSR = B01000111;
sei();}The line "WDTCSR = B01000111;" is doing two things. The B01000000 is closing changes to this register and the B00000111 is setting the time to 2 seconds. The flag WDTO_2S is 7 which in binary is B00000111.
For my own code I had to set the same register with this line:
WDTCSR = B01000000 | period;
The period was passed into the sleep function and was chosen from the list of flags in the wdt library header file:
0 = WDTO_15MS
1 = WDTO_30MS
2 = WDTO_60MS
3 = WDTO_120MS
4 = WDTO_250MS
5 = WDTO_500MS
6 = WDTO_1S
7 = WDTO_2S
8 = WDTO_4S
9 = WDTO_8S
The results.
 |
| My test rig, the clock is wired up in series with multi-meter measuring milli amps. |
 |
| After. |
 |
| Before. |
Improvements.
- Figure out why I could not use the 4 or 8 second delays with the old chip I am using. Waking up every 2 seconds uses 4 times the power of just waking up once every 8 seconds. Are these timings only usable on specific or newer chips? Does this need to be an option that someone can set if the bigger timings don't work for their platform?
- Get rid of the pointers for next/prev and instead just use a small array to track the tasks. Limit the number of tasks to less than 256 and we could just use byte pointers to the array for next/prev.
- The index i in the function that selects the period is always equal to the period value that is chosen. Just get rid of this column in the array to save memory and pass in the index as the period.
- Keep track of how much time we were in power down and just drag the time forward for the tasks once at the end. Add up each block of time in a variable and make the call with this variable once.
- Add different power down mode choices in the library to support various power saving features. One thing I would need to figure out is which modes need the time adjust for the tasks and which one the timer that millis() needs is still working. Would these various modes choices be compiled in? Or should it be done in runtime. The runtime option would give the arduino a dynamic processor control but at the cost of a larger runtime for the library.
- For the modes where the millis() timer still runs, fix the 50 day timer wrap problem. Basically everything needs pulled forward back to zero.
Sunday, March 13, 2016
Pihole, squid, squidguard on a $5 Pi Zero
My plan was to get the most use possible from the $5 raspberry Pi Zero that I managed to get my hands onto by buying a full kit that included a couple cables and a power supply. It was actually not that bad a deal. One pain that I was having is the fact that my mobile devices and android machines do not block ads. At all. Can't even load a plugin to the browser to block ads. Not to mention all the ads that are strewn through all the apps on the phones and pad computers. So when I saw Pihole again on reddit a couple of weeks ago, I knew that this was the project to use for my Pi Zero.
I first loaded raspberian lite onto a 16 GB class 10 SD card. I loaded this into a raspberry pi 2 for the initial setup. At this point you should set up a static ip address, either handed out by the router, or outside the range handed out by the router.
Also expand out the file system and set the gpu_mem=16MB in /boot/config.txt to free up as much memory as possible.
I loaded the script from https://pi-hole.net by copying it from the web page and pasting it into a terminal as root. At one point the install nuked /etc/resolve.conf and I had to add the line
The testing showed that it all worked. It worked to block about 18% of the links my browser was trying to load. It used less than 1% on the Pi Zero CPU for this and a tiny amount of memory. Web pages began loading in half the time.
I added a few local host names into the /etc/hosts file on the Pihole machine and reran gravity.sh command to reload everything, and then every machine was able to resolve those hostnames. This means that I no longer have to maintain a hosts table on all my clients, which is nice.
There is also a very nice admin console for the Pihole so you can see how effective it is. http://piholeserveraddress/admin/index.php
I powered down the RasPi 2 and put the card into the Pi Zero with an old 100MB usb network dongle attached to a usb on the go cable. I also swapped the 2 amp power supply with a smaller 1.5 amp power supply. The pi zero was in a small sleeve case. Everything worked just as fast with the $5 computer as it did with the $25 computer.
Watching the performance at less than 1% usage I realized that I could install a squid proxy server and found another ad blocker called squidguard that works as a pattern matching blocker. I had wanted a squid server for a decade now and had never gotten around to making one.
I followed this guide to set up squid and squid guard. The first thing you need to do is install, configure, and test squid.
Once that works you have to install, configure, tie into squid, restart squid, and test with a blacklist that just blocks a handful of sites.
Once that was done I downloaded the easylist that ad block plus uses, found a sed script that would translate easylist.txt into a blacklist that squidguard could use, and then created the blacklist.db file. I found the script and a write up of how to do that here. The script in the article does not work, but in the comments is a link to a script that does work on the github site.
At this point every device on my network is ad blocked. Phones, pad computers, both iphone and android, desktop machines, servers are all protected now, both browsers and apps, and are loading web pages twice as fast.
One last thing, I had to run a command on the blacklist.db file that was created to set the group and owner to proxy.
I installed Nagios following a guide I found on the internet.
The only thing I had to change to get this to work was adding
I first loaded raspberian lite onto a 16 GB class 10 SD card. I loaded this into a raspberry pi 2 for the initial setup. At this point you should set up a static ip address, either handed out by the router, or outside the range handed out by the router.
Also expand out the file system and set the gpu_mem=16MB in /boot/config.txt to free up as much memory as possible.
I loaded the script from https://pi-hole.net by copying it from the web page and pasting it into a terminal as root. At one point the install nuked /etc/resolve.conf and I had to add the line
nameserver 8.8.8.8to the bottom of the file. After this one problem I reran the script and finished up. I changed the DNS server that the router used and rebooted my desktop machine so it would get all the changes.
The testing showed that it all worked. It worked to block about 18% of the links my browser was trying to load. It used less than 1% on the Pi Zero CPU for this and a tiny amount of memory. Web pages began loading in half the time.
 |
| The grey case is the Pi Zero, the white plug is the usb network card, and the sd card is there for scale. |
There is also a very nice admin console for the Pihole so you can see how effective it is. http://piholeserveraddress/admin/index.php
I powered down the RasPi 2 and put the card into the Pi Zero with an old 100MB usb network dongle attached to a usb on the go cable. I also swapped the 2 amp power supply with a smaller 1.5 amp power supply. The pi zero was in a small sleeve case. Everything worked just as fast with the $5 computer as it did with the $25 computer.
Watching the performance at less than 1% usage I realized that I could install a squid proxy server and found another ad blocker called squidguard that works as a pattern matching blocker. I had wanted a squid server for a decade now and had never gotten around to making one.
I followed this guide to set up squid and squid guard. The first thing you need to do is install, configure, and test squid.
Once that works you have to install, configure, tie into squid, restart squid, and test with a blacklist that just blocks a handful of sites.
Once that was done I downloaded the easylist that ad block plus uses, found a sed script that would translate easylist.txt into a blacklist that squidguard could use, and then created the blacklist.db file. I found the script and a write up of how to do that here. The script in the article does not work, but in the comments is a link to a script that does work on the github site.
At this point every device on my network is ad blocked. Phones, pad computers, both iphone and android, desktop machines, servers are all protected now, both browsers and apps, and are loading web pages twice as fast.
One last thing, I had to run a command on the blacklist.db file that was created to set the group and owner to proxy.
cd /var/lib/squidguard/db
chown proxy blacklist.db
chgrp proxy blacklist.db
Nagios
Now that I have all this nifty infrastructure and I have all sorts of available space on the pihole I am adding in a monitor to check the hosts on my local network and to check on a few web servers I am responsible for.I installed Nagios following a guide I found on the internet.
The only thing I had to change to get this to work was adding
server.modules += ( "mod_alias" )
to the top of 10-nagios3.conf
Other than just getting a blank page like every other ad on the internet because of pihole, the only clue I had to this error was one line in the error.log for lighttpd that said
WARNING: unknown config-key: alias.url (ignored)
And it took an hour to fix that.
I reran gravity.sh after I installed Nagios3. But in hindsight I think it was another network error that made me think I needed to run this, but no harm done.
Tomorrow I am going to start monitoring some sites. :D
-- --
15 Mar 2016
Today I got pings happening to everything on my network except my iphone, which is up and down all the time, for some strange reason. Not even nmap can find it on the network when the iphone goes to sleep. My android phone and two pad computers are up all the time, even when the screen is in standby.
I created a hosts folder in /etc/nagios3/ and configured /etc/nagios/nagios.cfg to read that directory using a line that said
On the server side I had to add in some plugins to give me the check_nrpe command that the guide talked about. I used this guide to figure that out.
-- -- --
Services.
There were already checks for http and ssh built into the /etc/nagios3/conf.d/services_nagios2.cfg file. Turning these on for clients just involved adding the host name that you made above to the comma seperated with no spaces list in hostgroups_nagios2.cfg
Where I found the smb script. Change line 11 to where basepath is on your system.
https://exchange.nagios.org/directory/Plugins/System-Metrics/File-System/SMB/check_smb/details
You have to put check_smb in /usr/lib/nagios/plugins/
Configure a file to tie the plugin to a nagios command here: /etc/nagios-plugins/config/smb.cfg
This is what I put in the file:
# check that smb services are running
--
Where I found raspberry pi logos for nagios.
https://exchange.nagios.org/directory/Graphics-and-Logos/Images-and-Logos/Raspberry-Pi/details
put that directory here:
/usr/share/nagios3/htdocs/images/logos
edit nagios to use them in the /etc/nagios3/conf.d/extinfo_nagios2.cfg file.
---
Tomorrow I get nrpe client running against a few hosts.
-- --
15 Mar 2016
Today I got pings happening to everything on my network except my iphone, which is up and down all the time, for some strange reason. Not even nmap can find it on the network when the iphone goes to sleep. My android phone and two pad computers are up all the time, even when the screen is in standby.
I created a hosts folder in /etc/nagios3/ and configured /etc/nagios/nagios.cfg to read that directory using a line that said
cfg_dir=/etc/nagios3/hoststhen I used this template:
define host { host_name hostname alias plainenglishname address 192.168.0.xxx max_check_attempts 3 check_period 24x7 check_command check-host-alive contacts root notification_interval 60 notification_period 24x7}
placing the files in the /etc/nagios3/hosts directory. The filename doesn't matter, but I used ipaddress-hostnam.cfg for all these files.
I had one problem on localhost for the pihole box running the nagios3 localhost disk monitor command. I had to do these commands to give it permission to work correctly.
chmod 755 /sysI worked on setting up nagios agent on one of my ubuntu desktop machines, following this pdf guide, doing the following:
chmod 755 /sys/kernel/
chmod 755 /sys/kernel/debug/
chmod 755 /sys/kernel/debug/tracing/
add-apt-repository ppa:nagiosinc/ppaThis adds a client that will let the nagios3 server run a lot of commands on remote machines.
echo "deb http://ppa.launchpad.net/nagiosinc/ppa/ubuntu lucid main" > /etc/apt/sources.list.d/nagiosinc.list
gpg --keyserver keyserver.ubuntu.com --recv-keys B18637BB5175BC68
gpg --export --armor B18637BB5175BC68 | apt-key add -
apt-get update
apt-get install nagios-agent
On the server side I had to add in some plugins to give me the check_nrpe command that the guide talked about. I used this guide to figure that out.
-- -- --
Services.
There were already checks for http and ssh built into the /etc/nagios3/conf.d/services_nagios2.cfg file. Turning these on for clients just involved adding the host name that you made above to the comma seperated with no spaces list in hostgroups_nagios2.cfg
Where I found the smb script. Change line 11 to where basepath is on your system.
https://exchange.nagios.org/directory/Plugins/System-Metrics/File-System/SMB/check_smb/details
You have to put check_smb in /usr/lib/nagios/plugins/
Configure a file to tie the plugin to a nagios command here: /etc/nagios-plugins/config/smb.cfg
This is what I put in the file:
# 'check_smb' command definitiondefine command{ command_name check_smb command_line /usr/lib/nagios/plugins/check_smb -H '$HOSTADDRESS$' }In /etc/nagios3/conf.d/services_nagios2.cfg file append these lines at the end:
# check that smb services are running
define service { hostgroup_name smb-servers service_description SMB check_command check_smb use generic-service notification_interval 0 ; set > 0 if you want to be renotified}
Then you can tie hosts to that service in /etc/nagios3/conf.d/hostgroups_nagios2.cfg with a block of text added to the end like this:
# A list of your smb-accessible serversdefine hostgroup { hostgroup_name smb-servers alias SMB servers members serverA,server2,serverIII }
And tie it to the hosts you want to run the check against on the members line. remember that these hosts all have to be predefined.
--
Where I found raspberry pi logos for nagios.
https://exchange.nagios.org/directory/Graphics-and-Logos/Images-and-Logos/Raspberry-Pi/details
put that directory here:
/usr/share/nagios3/htdocs/images/logos
edit nagios to use them in the /etc/nagios3/conf.d/extinfo_nagios2.cfg file.
 |
| My hosts list. |
 |
| The three services I am currently looking for, smb, http, and ssh. |
---
Tomorrow I get nrpe client running against a few hosts.
Subscribe to:
Posts (Atom)