Sunday, March 27, 2016

My blog has broken 50,000 page views. :D

My blog passed 50,000 page views today.  I want to thank everyone who has visited and helped me pass this goal.  



Saturday, March 26, 2016

Low Power Scheduling Library for Arduino 328U

I wrote my own scheduling library for Arduino a couple of years ago. One of the to-do's for that project was to replace the main loop with a sleep/power down feature that would reduce power down but still accurately wake up and perform the tasks when required.

The reasons that you want to use less power for a microcontroller project is easy.  You want your project to run longer on a set of batteries.  You want your solar powered project to work with the smaller, cheaper solar panel.  You want your coin cell powered camera trigger to work for years, instead of months.

Last weekend I sat down to actually program the power saving feature that I had imaged a couple of years ago. My first few dozen tries failed, and on Sunday of last weekend all I had working was to put the processor in idle mode.  Which I took as a win, because at least the sleep functions of my program were working.

I read a lot of various web pages about how to use the watch dog timer on Arduino, but the one that actually let me figure it out was the post that starts with "OK, I think I have it" on this web page.  I used that example program as a simple example that worked to figure out what I was doing wrong.  I found that example this weekend, after poking at things all week long and going to sleep thinking about the problem.

The first draft of my code that actually worked was just uploaded to github. There were just 2 tricky changes I had to make to my library code.

The first one was the fact that the watchdog timer can't be set to trigger at any arbitrary length of time.  It only triggers at discrete units of time, sort of like a quantum energy emission.  So I put a loop in that found the largest block of time that could fit in the scheduled time, and went to sleep for that amount of time, then it would wake up when the watch dog timer fired and find the next largest remaining block of time that could fit in the delay we needed.  If there is a little bit of time left that is less than the 15ms minimum time that is left, I just delay.

The second thing was the fact that my code requires the use of the millis() function call, but that the timer in the micro-processor that keeps track of time is suspended when the chip is in power down mode.  So to fix this I just drag all the scheduled tasks ahead by the amount of time that I just slept. A side effect of this is that because this timer almost never runs, I think I fixed the 50 day timer wrap problem. :D

One of the most difficult things to figure out was that for some reason the function to set the watch dog timer delay was not working, you had to set the register directly.  From the example code that helped me figure this out.

void myWatchdogEnable() {    
// turn on watchdog timer;   
// interrupt mode every 2.0s cli();  
MCUSR = 0;  
WDTCSR |= B00011000;  
WDTCSR = B01000111;  
sei();}
The line "WDTCSR = B01000111;" is doing two things.  The B01000000 is closing changes to this register and the B00000111 is setting the time to 2 seconds.    The flag WDTO_2S is 7 which in binary is B00000111.

For my own code I had to set the same register with this line:
WDTCSR = B01000000 | period; 
The period was passed into the sleep function and was chosen from the list of flags in the wdt library header file:
0 = WDTO_15MS
1 = WDTO_30MS
2 = WDTO_60MS
3 = WDTO_120MS
4 = WDTO_250MS
5 = WDTO_500MS
6 = WDTO_1S
7 = WDTO_2S
8 = WDTO_4S
9 = WDTO_8S
The results.

And because Science requires documentation, I used my multi-meter to measure the difference between my old and new scheduling libraries using my clock sketch.

My test rig, the clock is wired up in series with multi-meter measuring milli amps.

After.
Before.


As you can see, the change is nearly a 50% power savings. And this is not even optimizing the hardware being used to reduce wasted current.  How would you like your Arduino project to run for 18 hours instead of 10 hours on one charge?

Improvements.


  • Figure out why I could not use the 4 or 8 second delays with the old chip I am using.  Waking up every 2 seconds uses 4 times the power of just waking up once every 8 seconds.  Are these timings only usable on specific or newer chips?  Does this need to be an option that someone can set if the bigger timings don't work for their platform?

  • Get rid of the pointers for next/prev and instead just use a small array to track the tasks. Limit the number of tasks to less than 256 and we could just use byte pointers to the array for next/prev.

  • The index i in the function that selects the period is always equal to the period value that is chosen.  Just get rid of this column in the array to save memory and pass in the index as the period.

  • Keep track of how much time we were in power down and just drag the time forward for the tasks once at the end.  Add up each block of time in a variable and make the call with this variable once.

  • Add different power down mode choices in the library to support various power saving features.  One thing I would need to figure out is which modes need the time adjust for the tasks and which one the timer that millis() needs is still working. Would these various modes choices be compiled in? Or should it be done in runtime.  The runtime option would give the arduino a dynamic processor control but at the cost of a larger runtime for the library.

  • For the modes where the millis() timer still runs, fix the 50 day timer wrap problem.  Basically everything needs pulled forward back to zero.



Sunday, March 13, 2016

Pihole, squid, squidguard on a $5 Pi Zero

My plan was to get the most use possible from the $5 raspberry Pi Zero that I managed to get my hands onto by buying a full kit that included a couple cables and a power supply.  It was actually not that bad a deal.  One pain that I was having is the fact that my mobile devices and android machines do not block ads.  At all.  Can't even load a plugin to the browser to block ads.  Not to mention all the ads that are strewn through all the apps on the phones and pad computers. So when I saw Pihole again on reddit a couple of weeks ago, I knew that this was the project to use for my Pi Zero.

I first loaded raspberian lite onto a 16 GB class 10 SD card.  I loaded this into a raspberry pi 2 for the initial setup.  At this point you should set up a static ip address, either handed out by the router, or outside the range handed out by the router.

Also expand out the file system and set the gpu_mem=16MB in /boot/config.txt to free up as much memory as possible.

I loaded the script from https://pi-hole.net by copying it from the web page and pasting it into a terminal as root.  At one point the install nuked /etc/resolve.conf and I had to add the line
nameserver 8.8.8.8
to the bottom of the file. After this one problem I reran the script and finished up.  I changed the DNS server that the router used and rebooted my desktop machine so it would get all the changes.

The testing showed that it all worked. It worked to block about 18% of the links my browser was trying to load. It used less than 1% on the Pi Zero CPU for this and a tiny amount of memory. Web pages began loading in half the time.

The grey case is the Pi Zero,
the white plug is the usb network card,
and the sd card is there for scale.

I added a few local host names into the /etc/hosts file on the Pihole machine and reran gravity.sh command to reload everything, and then every machine was able to resolve those hostnames.  This means that I no longer have to maintain a hosts table on all my clients, which is nice.

There is also a very nice admin console for the Pihole so you can see how effective it is.  http://piholeserveraddress/admin/index.php



I powered down the RasPi 2 and put the card into the Pi Zero with an old 100MB usb network dongle attached to a usb on the go cable.  I also swapped the 2 amp power supply with a smaller 1.5 amp power supply.  The pi zero was in a small sleeve case. Everything worked just as fast with the $5 computer as it did with the $25 computer.

Watching the performance at less than 1% usage I realized that I could install a squid proxy server and found another ad blocker called squidguard that works as a pattern matching blocker. I had wanted a squid server for a decade now and had never gotten around to making one.

I followed this guide to set up squid and squid guard. The first thing you need to do is install, configure, and test squid.

Once that works you have to install, configure, tie into squid, restart squid, and test with a blacklist that just blocks a handful of sites.

Once that was done I downloaded the easylist that ad block plus uses, found a sed script that would translate easylist.txt into a blacklist that squidguard could use, and then created the blacklist.db file. I found the script and a write up of how to do that here. The script in the article does not work, but in the comments is a link to a script that does work on the github site.

At this point every device on my network is ad blocked.  Phones, pad computers, both iphone and android, desktop machines, servers are all protected now, both browsers and apps, and are loading web pages twice as fast.

One last thing, I had to run a command on the blacklist.db file that was created to set the group and owner to proxy.
cd /var/lib/squidguard/db
chown proxy blacklist.db
chgrp proxy blacklist.db 

Nagios

Now that I have all this nifty infrastructure and I have all sorts of available space on the pihole I am adding in a monitor to check the hosts on my local network and to check on a few web servers I am responsible for.

I installed Nagios following a guide I found on the internet.

The only thing I had to change to get this to work was adding
server.modules += ( "mod_alias" )
to the top of 10-nagios3.conf 

Other than just getting a blank page like every other ad on the internet because of pihole, the only clue I had to this error was one line in the error.log for lighttpd that said
WARNING: unknown config-key: alias.url (ignored)
And it took an hour to fix that.

I reran gravity.sh after I installed Nagios3.  But in hindsight I think it was another network error that made me think I needed to run this, but no harm done.

Tomorrow I am going to start monitoring some sites. :D

-- --

15 Mar 2016

Today I got pings happening to everything on my network except my iphone, which is up and down all the time, for some strange reason.  Not even nmap can find it on the network when the iphone goes to sleep. My android phone and two pad computers are up all the time, even when the screen is in standby.

I created a hosts folder in /etc/nagios3/  and configured /etc/nagios/nagios.cfg to read that directory using a line that said
cfg_dir=/etc/nagios3/hosts
then I used this template:
define host {  host_name hostname  alias plainenglishname  address 192.168.0.xxx   max_check_attempts 3  check_period 24x7  check_command check-host-alive  contacts root  notification_interval 60  notification_period 24x7}
placing the files in the /etc/nagios3/hosts directory. The filename doesn't matter, but I used ipaddress-hostnam.cfg for all these files.

I had one problem on localhost for the pihole box running the nagios3 localhost disk monitor command.  I had to do these commands to give it permission to work correctly.  
chmod 755 /sys
chmod 755 /sys/kernel/
chmod 755 /sys/kernel/debug/
chmod 755 /sys/kernel/debug/tracing/
I worked on setting up nagios agent on one of my ubuntu desktop machines, following this pdf guide, doing the following:
  add-apt-repository ppa:nagiosinc/ppa
  echo "deb http://ppa.launchpad.net/nagiosinc/ppa/ubuntu lucid main" > /etc/apt/sources.list.d/nagiosinc.list
  gpg --keyserver keyserver.ubuntu.com --recv-keys B18637BB5175BC68
  gpg --export --armor B18637BB5175BC68 | apt-key add -
  apt-get update
  apt-get install nagios-agent
This adds a client that will let the nagios3 server run a lot of commands on remote machines.

On the server side I had to add in some plugins to give me the check_nrpe command that the guide talked about.  I used this guide to figure that out.

-- -- --

Services.

There were already checks for http and ssh built into the /etc/nagios3/conf.d/services_nagios2.cfg file.  Turning these on for clients just involved adding the host name that you made above to the comma seperated with no spaces list in hostgroups_nagios2.cfg


Where I found the smb script.  Change line 11 to where basepath is on your system.

https://exchange.nagios.org/directory/Plugins/System-Metrics/File-System/SMB/check_smb/details

You have to put check_smb in  /usr/lib/nagios/plugins/

Configure a file to tie the plugin to a nagios command here: /etc/nagios-plugins/config/smb.cfg

This is what I put in the file:
# 'check_smb' command definitiondefine command{ command_name check_smb command_line /usr/lib/nagios/plugins/check_smb -H '$HOSTADDRESS$' }
In /etc/nagios3/conf.d/services_nagios2.cfg file append these lines at the end:

# check that smb services are running
define service {        hostgroup_name                  smb-servers        service_description             SMB check_command                   check_smb        use                             generic-service notification_interval           0 ; set > 0 if you want to be renotified}
Then you can tie hosts to that service in /etc/nagios3/conf.d/hostgroups_nagios2.cfg  with a block of text added to the end like this:

# A list of your smb-accessible serversdefine hostgroup {        hostgroup_name  smb-servers alias           SMB servers members         serverA,server2,serverIII        }

And tie it to the hosts you want to run the check against on the members line. remember that these hosts all have to be predefined. 


--

Where I found raspberry pi logos for nagios.

https://exchange.nagios.org/directory/Graphics-and-Logos/Images-and-Logos/Raspberry-Pi/details

put that directory here:

/usr/share/nagios3/htdocs/images/logos

edit nagios to use them in the /etc/nagios3/conf.d/extinfo_nagios2.cfg file.

My hosts list.

The three services I am currently looking for, smb, http, and ssh.


---

Tomorrow I get nrpe client running against a few hosts.



Tuesday, February 23, 2016

Using Arduino to Test Network Cables.

Using 2 network connectors that normally snap into wall plates and accept network cables I built a network cable tester using an Arduino Pro Mini, a breadboard, a couple of inches of network cable and 8 1M Ohm resistors. The code I used is at the end of the page.  And off course, if you wire things differently, your pinout order might have to change.

I wired up one of the network cable connectors to just turn the data around.  I looked at A mode and just hooked green up to green white, orange up to orange white, blue up to blue white, and brown up to brown white.  The two punch downs on the side closest to where the wire connects just loop around, and the two on the back side connect strait across.

The other end I wired up to the A mode as well, using a 3 inch piece of network cable.  I followed the same color code on this side, the A mode, and matched the color of the cable to the little block of color on the side of the connector to the color of the wire.  I stripped the ends of the wires and pushed them into the breadboard so they lined up with pins 2 through 9.

Then I connected one end of a cable to the turn around plug and the other end into the connector attached to the Arduino.  At this point I thought I was pretty much done.  So I wrote the program and immediately just began to get random data from the pins.  Seems that if they are set to input they will float and hold a voltage like a capacitor for many minutes after they had been charged up. And even waving your hand near them is enough to make them read high. So I had to add 8 high value resistors from each pin to ground to bleed off this voltage, and the once nice and neat breadboard looks very messy.  What I need is a high value resistor pack that will just plug in to all those pins and to ground. This allowed excess voltage to bleed off over about 10 milliseconds time and let me read the voltage correctly.

The code was pretty strait forward.  I just needed to set all the pins into input mode and then one at a time set a pin to output mode HIGH and then read all the other pins to make sure I get a voltage on one and only one pin, and that it is the correct pin.   Then set that pin back to LOW and make it an input again, and select the next pin and do the same thing.  If everything is good, light an LED green.

The Arduino is able to easily perform these checks 4 times a second, giving me the opportunity to move the cables around and see of there are any intermittent contacts on either end or breaks in the wire.

I found the correct pin order by reading a known good cable.  It was an A cable, then I read a B cable and it returned the exact same pin order.  So I thought that a cross over cable must be different, but it returned the exact same pin order as A and B modes.  So that made that easy. Still don't understand why a cross over cable works the same as A and B. I tested it on several cables to make sure the cables I had were correct.

The code is as follows:


//Uncomment the following line to see output on the serial terminal.
//#define SERIALDEBUG 1

int good[8]= { 3, 2, 5, 4, 7, 6, 9, 8};

void setup() {                
#ifdef SERIALDEBUG
  Serial.begin(115200);
  //Serial.println("Checking pins.");
#endif
  
  // initialize the digital pin as an output.
  pinMode(12, OUTPUT);
}

void loop() {
  
  int x, y, z;
  // Set all the pins low.
  for (x=2; x<10; x++) {
    digitalWrite(x, LOW); 
    pinMode     (x, INPUT);
  }
  
  z=0;
  // Go through each pin, one at a time.
  // find where pin was routed to
  for (x=2; x<10; x++) {
    
    // setup current pin
    pinMode(x, OUTPUT);
    digitalWrite(x, HIGH); 
    delay(10); 

    // Find matching pin.
    for (y=2; y<10; y++) {

       if (x==y) continue;
       if (digitalRead(y) == HIGH) {
         if (good[x-2] == y)
            z++;
         else
            z+=100;
#ifdef SERIALDEBUG            
         Serial.print("good ");
         Serial.print(good[x-2]);
         Serial.print(" Z ");
         Serial.print(z);
         Serial.print(" Pin ");
         Serial.print(x);
         Serial.print(" is wired to pin ");
         Serial.println(y);
#endif
       }
    }
    
    // Set current pin back to the rest
    digitalWrite(x, LOW);
    pinMode     (x, INPUT);
  }
  
  if (z == 8)
      digitalWrite(12, HIGH);
  else
    digitalWrite(12, LOW);

#ifdef SERIALDEBUG
  Serial.println("");
  delay(1000);
#else
  delay(200);
#endif
}

Rasberry PI cluster in a Stackable Tube.

Getting down a bunch of ideas I have been having about building a raspberry pi cluster. I got this idea from reading how cray computers used to be built in a circle to reduce the length of the wires that connected the individual computers together. Evidently this resulted in significant measurable speed ups in message passing latency.

Picture 12 raspberry pi computers in a circle with their GPIO pins on the inside, their hdmi slots facing out, and their memory cards facing up. very short wires from each board going to every other board. Serial networking protocols running on those serial TTL connections, eliminating the need for more hardware and reducing latency to very small values.   Disadvantage would be low bandwidth on any one network segment.

One of the Pi computers acts as the master and connects to local network with a standard network cable and routes to the rest of the computer
s in the cluster.  Alternatively you could run network connections to every computer and use this for high latency, high bandwidth data.

Doing the low latency network connections would speed up the cluster if it was processing many tiny messages that need processed as quickly as possible, such as financial data, or medical HL7 events.

A single brick power supply sends power directly into the GPIO of each board, bypassing all the safety measures on the boards, so the power supply would have to be a good one. This eliminates the need for running a usb cable to each individual board for power.

Right now I am working on printing out tube segments that will fit on top of each other and hold the PIs. These tubes would be passively cooled, the tube acting like a chimney. So it would have to be on a base that allowed air flow from underneath. But also design them to be stacked with a connecting segment on each one with the middle clear so you can drop power and network cables down the middle. This connecting segment would pull in air from the front, let it go up the tube above it, while letting the air from the tube below it go behind the stack of tubes. If need be a small fan on the back of each stacking segment could pull air up through each segment.

Alternatively the tubes could stack on top of each other and a single 120mm fan on the bottom rapidly blows air up through all the tubes.

A topper segment neatly closes the top off from view, while allowing air to blow up easily. The base could accept legs or screw down to a board for better stability. Or the whole thing could hang from a ceiling beam.

With Pi Zeros the cluster would only draw about 7 watts max. With Raspi 2s it could draw up to 25 watts and be 3 times more expensive, but it would be almost an order of magnitude more capable. Power data comes from here. Nodes can power down unused parts of their board for more savings.

A clear plastic tube over them proudly showing the details and directing cooling air flow. This could be done by cutting the tubing and pressure fitting them into 3D printed fixtures at the top and bottom. A 3D printed frame fits into the clear tube and the tube goes into rings top and bottom to make it look finished.

Saturday, January 30, 2016

An Inexpensive Marketing Kiosk.


This page is under construction.  This is my disorganized mind just trying to throw all my ideas at the wall and see which ones stick.  Right now I am just figuring it out myself, so any suggestions would be awesome.  

The problem I am trying to solve is that my company has several former desktop computers each playing a fixed playlist of marketing videos around the store.  It is tough to manage these computers and most weeks the store staff just forgets to even turn them on.  Plus they are power hogs.  So, what I want is something that can be programmed by just plugging in a USB key and that is low cost,  both to make and to operate.  

This would be useful for just about any organization that needs to answer frequently asked questions  about their organization and what it does and how to do common tasks. It could be used at a job fair to tell about a company, in a store to give product information, in the lobby of a bank or company to give an overview about the company. 

In this post I am going to describe how to make a display that sits on a table and allows someone interested in your product or organization to play videos, audio content, or slide shows of their choosing from a list.

Steps of setup

  1. Put box on table.
  2. Plug in box.
  3. Box boots and looks for usb drive with files on it.  
  4. If USB thumb drive not found, then show how to setup video.
  5. If found, then begin operation mode as follows.
Programming the device should be as easy as plugging in a USB key with audio, video, or slide shows in Microsoft powerpoint or Libre Office format. The title of the content is what is presented on the screen to select it.

Steps of operation

  1. Person walks up to kiosk.
  2. Motion sensor detects person.
  3. Monitor is taken out of standby.
  4. A Short "Please touch one of the buttons to select a presentation, or press the up and down to navigate to other selects" plays.
  5. Menu Screen: Present titles on monitor next to each of the 8 buttons, 4 on each side of the monitor.
  6. Person pushes a button and presentation plays, then returns back to the menu screen.
  7. If any button is pressed while the presentation is playing, the presentation stops and returns to the menu screen.  
  8. After a few minutes of not sensing any motion, after the video is done playing, put monitor back to sleep and wait.

What we need:

8 joystick buttons with micro-switches. 
1 hdmi monitor.
1 $5 Raspberry PI Zero, or any of the other low cost systems running Linux.
1 USB hub.  
1 extension cord with at least 2 outputs.
a lot of Foam board
1 wireless keyboard and mouse, to manage the device, can be reused on other kiosks.
1 motion detector.

So, what we need to do is make a box out of foam board that is strong enough to support the monitor and switches and heavy enough to not get bumped off the table.  This could be made free standing if need be, but I would build a wooden frame inside the foam board if it has to be freestanding and put a couple of concrete blocks in the bottom. 

Make sure when you are planning to make the foam core or cardboard box that you ensure that everything will fit inside the box.  Make sure that where you are placing the  

Once the box is made, carefully mark where the monitor and push buttons will go, making sure that there is free space for everything and that  nothing is  too close to anything else.  Then cut them out a little tight and dry fit everything. 

The raspi will mount to the back of the monitor using the VESA mounts. The monitor will be fitted into the opening about halfway and taped in place all around with clear packing tape.  Be sure not to block any vent holes. The top of the box and a space in back should be left open to allow air to circulate.  The extension cord will be zip tied along the back of the box with the cord extending out of the box. 

Once the kiosk box has all the hardware securely fitted inside, then you plug a 5v power supply up for the raspi and plug the monitor in. If needed add a 120mm fan and a power supply to blow hot air up out of the box. 

Work that is left to do:


Connecting a push button to a Raspberry PI.  We will need to do this for all 8 arcade style push buttons.

http://razzpisampler.oreilly.com/ch07.html


Connecting the PIR motion sensor to a Raspberry PI.

http://pimylifeup.com/raspberry-pi-motion-sensor/


Programming the code on the machine to work the magic. 

A python program should be able to check things out for us and use a state machine to figure out what to do and when following the two checklists above on setup and operation. 

We basically also just want to run X and then take over the screen and manage it all ourselves. An auto login to an account that runs our custom code would be awesome, as well as being able to log out and go into a separate admin account to program the kiosk. 

Friday, November 20, 2015

Time to end the cloud computing model.

The cloud is not your friend.  There are security and surveillance issues that make using a 3rd party system untenable for anyone that cares about their privacy. And on top of the government monitoring everything we do on the internet, corporations freely consolidate everything we do as well, recording nearly every mouse movement and keystroke we do on social media sites for big data analysis.

What we need to do is to pull our data and our processing of that data back onto computer systems that we control, so that we can decide what we share and with whom.  If the government wants our data, they can get a warrant and ask us for the data.  Even though you must hand the data over in either case, at least with a warrant the data can only be used for a specific purpose, in support of already existing probable cause, and not be used as a giant fishing expedition.

I propose we use the raspberry pi version 2, or similar system, as a computing platform for this endeavor.  Even smart phones are getting good enough to support such a project.  We need to build a set of web applications running on the box facing the Internet that will provide a way to securely and directly share emails, messages, social media posts, voice, and video with a set of friends.  No stops on a central server for the data, so there would be no ability for that service to be required to collect the data and hand it over en masse, and no ability to screen the data and data mine it for personal information.  Because only a few hundred friends at most would be hitting these sites then it would be low volume.

From your own point of view you would see a facebook or GPlus like web site of a list of posts and messages from your friends.  It would be possible to have dozens of different apps to view this data in many different ways, and the apps each could allow user customization of many features.

You could read and comment on these posts and the comments would flow back to your friends site in the background and then back out to all their friends.  You could record a voice or video message for a friend to pick up later, or you could directly connect with messaging, voice, or video if both of you are online.

The one weakness to this model that I see is that you may still need a central directory that everyone can register with when the box comes online to find your friends current IP address before you can directly connect with them.  But a central site could be compromised and the meta data about the boxes connected could be leaked with just a sealed letter from the government.

Each time you connect to the internet your ip address can change. As long as just one boxes ip address changes then it could easily just connect back up to the previous ip addresses all their friends. We may need to use mutual friends whose address has not changed to reconnect two people whose ip addresses have both changed.

A way to dynamically update a dns address on a server could work, and use the existing infrastructure for the name to ip address resolution.  Anyone logging DNS requests could build up a list of your friends from the names you request.