Atik Cameras

Author Topic: AtikAirService & ArtemisHScService crashes  (Read 12562 times)

razor

  • Newbie
  • *
  • Posts: 5
AtikAirService & ArtemisHScService crashes
« on: November 19, 2016, 07:13:36 AM »
Is there a way to monitor an output log of these services?  I'm getting constant crashes on my raspberry PI while running a few exposures (8s+) and then stopping it.  Is there a way to gracefully recover? Also, has the AtikAir image (which I am using) been updated? How do we keep our PI up to date without re-imaging?

I also see some startup problems, see the daemon log here:

Nov 19 07:07:43 raspberrypi systemd[1]: ArtemisHscService.service start operation timed out. Terminating.
Nov 19 07:07:43 raspberrypi systemd[1]: AtikAirService.service start operation timed out. Terminating.
Nov 19 07:07:43 raspberrypi systemd[1]: Failed to start (null).
Nov 19 07:07:43 raspberrypi systemd[1]: Unit AtikAirService.service entered failed state.
Nov 19 07:07:43 raspberrypi systemd[1]: Failed to start (null).
Nov 19 07:07:43 raspberrypi systemd[1]: Unit ArtemisHscService.service entered failed state.


When it does work, it works like a dream though. It's awesome.  Just frustrated with all the crashes, I can't seem to get past maybe 3-5 minutes without crashiing.


razor

  • Newbie
  • *
  • Posts: 5
Re: AtikAirService & ArtemisHScService crashes
« Reply #1 on: November 19, 2016, 07:37:05 AM »
Here's some more details from the daemon log. Looks like the send fails mid stream.

Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: -- StartExposureReceived
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Start Exposure: 57 8.00ms (1253612)
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Image NOT Ready
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Image NOT Ready
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: --Switch to Second
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: ExposureLength: 2 0 8000 0 8 333
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Exposure Downloaded
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Image Ready
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Image Ready
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Big Message! 2895423
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Send100ms: 1295423
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: ExposureThreadTime 8375
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Big Message Sent!!
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Big Message! 2895423
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Send100ms: 1295423
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: --SEND FAILED!!
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Send100ms: 1295423
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: --SEND FAILED!!
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Send100ms: 1295423
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: --SEND FAILED!!
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Send100ms: 1295423
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: --SEND FAILED!!
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Send100ms: 1295423
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: --SEND FAILED!!
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Send100ms: 1295423
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: --SEND FAILED!!
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Send100ms: 1295423
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Big Message Sent!!
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: --SEND FAILED!!
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: --SEND FAILED!!
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: --SEND FAILED!!

Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: ProcessMessage: 1201
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: ProcessMessage: 401
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: -- StartExposureReceived
Nov 19 07:35:23 raspberrypi ArtemisHscService[488]: Start Exposure: 58 8.00ms (1265219)


ChrisGolden

  • Atik Staff
  • Full Member
  • *****
  • Posts: 161
Re: AtikAirService & ArtemisHScService crashes
« Reply #2 on: November 21, 2016, 01:37:50 PM »
Hi Razor,

Sorry to hear you're having problems. If the messages are failing to send, the most likely reason is that the connection between the PI and PC is no longer open. Is it possible your wifi is dropping out? That won't explain the service not starting up properly though. How did you set up your raspberry pi? Using the image or the advanced setup?

Chris

razor

  • Newbie
  • *
  • Posts: 5
Re: AtikAirService & ArtemisHScService crashes
« Reply #3 on: November 21, 2016, 07:17:03 PM »
I used the image to deploy the PI.  Would you recommend re-deploying from the latest image you linked to dropbox?

ChrisGolden

  • Atik Staff
  • Full Member
  • *****
  • Posts: 161
Re: AtikAirService & ArtemisHScService crashes
« Reply #4 on: November 22, 2016, 02:11:25 PM »
Hi Razor,

So far there has only been one AtikAir image. I have made some changes to the AtikAir app on the PC side (Which are included in the latest installer) but these wouldn't account for the issue you are having. A few thoughts: Are you using a raspberry pi 3? And have you made any changes to pi after installing the image? Also, assuming the service starts up properly, if you then run with the pi still connected via an ethernet cable (and the PC pointed at the ethernet connection), does it work OK?

Chris.

ivygda

  • Newbie
  • *
  • Posts: 2
Re: AtikAirService & ArtemisHScService crashes
« Reply #5 on: November 23, 2016, 12:47:38 AM »
Hi Razor, Chris

I've been following this thread with interest and may have some useful info here.  But before going into details, first and foremost I would like to thank Chris and the team at Atik for going down the AtikAir path and putting together some first rate software to help us do our Astrophotography. The detailed analysis that follows is provided purely to help flush out the last few little bugs and get everything really robust.

I recently set up a Pi 3 B 1GB RAM 8GB SD card with AtikAir (image method) and connected it via a good cat6 Ethernet cable to a managed switch. Initially I just used it exactly as specified in the instructions - no mods. It didn't work even though the router had given it an address via DHCP (192.168.1.139) and logged it offering the name 'raspberrypi' .

The AtikAir App (recently installed via 4.1.0.9 setup on a PC with very little software on it) didn't find it. Even if I used the Set IP option, the Remote button showed 'Not Available'. I tried ArtemisCapture and it couldn't connect. Also tried the App from the PC that normally drives my scope - lots of stuff connected and a horribly messy software environment but it behaved the same (also setup 4.1.0.9)

I connected to the Pi via ssh using PuTTY from a PC and found a few things to go after.

/var/log/daemon.log shows the services AtikAirService and ArtemisHscService falling over as Razor found. I looked for possible causes, focusing on what might be different between the development environment and homed in on the network setup - as noted by other threads.

/etc/network/interfaces referenced two wifi adaptors with manual addressing, both to be configured from /net/wpa_supplicant/wpa_supplicant by wpa-conf ...but that file was missing - presumably deleted to avoid giving out the wifi password at development location. I edited /net/interfaces to just have the one wireless adaptor using dhcp and put in a wpa_supplicant file using WPA2 with SSID and password to match and rebooted. The router renewed the old lease 1.139 for the copper and gave out 1.30 for the wifi. Fortunatley the Pi's Linux setup seems very tolerant of multiple network interfaces and doesn't seem to need careful routing setup to live like that.

Went back to AtikAir app ... and still couldn't find the Pi... and Remote still shows 'Not Available' BUT ... Artemis Capture finds the camera (recent Atik 460) and can connect. I expected to find the AtikAirService and ArtemisHscService services now stable but they still failed 5 mins after startup - or at least they make that entry in the logs. Strange. Here's the status with log extract via systemctl:

root@raspberrypi:/var/log# systemctl status -l ArtemisHscService.service
● ArtemisHscService.service - (null)
   Loaded: loaded (/etc/init.d/ArtemisHscService)
   Active: failed (Result: timeout) since Sun 2016-11-20 16:02:51 UTC; 2 days ago
  Process: 487 ExecStart=/etc/init.d/ArtemisHscService start (code=killed, signal=TERM)

Nov 22 23:47:16 raspberrypi ArtemisHscService[487]: Send100ms: 890165
Nov 22 23:47:16 raspberrypi ArtemisHscService[487]: Big Message Sent!!
Nov 22 23:47:16 raspberrypi ArtemisHscService[487]: ProcessMessage: 401
Nov 22 23:47:16 raspberrypi ArtemisHscService[487]: -- StartExposureReceived
Nov 22 23:47:16 raspberrypi ArtemisHscService[487]: Start Exposure: 1912 240.00ms (200558442)
Nov 22 23:47:16 raspberrypi ArtemisHscService[487]: Image NOT Ready
Nov 22 23:47:16 raspberrypi ArtemisHscService[487]: --Switch to First
Nov 22 23:47:16 raspberrypi ArtemisHscService[487]: ProcessMessage: 1202
Nov 22 23:47:16 raspberrypi ArtemisHscService[487]: ProcessMessage: 1204
Nov 22 23:47:16 raspberrypi ArtemisHscService[487]: ProcessMessage: 1301
root@raspberrypi:/var/log#


HOWEVER ArtemisCapture works well now and robustly captures images - more on this in a second. Looking for the daughter processes from the service scripts - they are still up:

# ps -ax
  477 ?        Ss     0:00 /sbin/wpa_supplicant -s -B -P /run/wpa_supplicant.wlan0.pid -i wlan0 -D nl80211,wext -c /etc/wpa_supplicant/wpa_supplicant.conf
  496 ?        Sl     0:49 /usr/lib/artemishscservice
  497 ?        S      0:00 /usr/lib/atikairservice
  584 ?        S<     0:00 [kworker/3:1H]
  662 ?        S<     0:00 [kworker/u9:0]
  663 ?        S<     0:00 [hci0]
  664 ?        S<     0:00 [hci0]
  665 ?        S      0:00 /usr/bin/hciattach /dev/serial1 bcm43xx 921600 noflow -
  666 ?        S<     0:00 [kworker/u9:1]
  670 ?        Ss     0:00 /usr/lib/bluetooth/bluetoothd
  815 ?        Ss     0:00 /sbin/dhcpcd -q -w
  816 ?        Ss     0:00 /usr/sbin/sshd -D
  832 ?        Ss     0:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 106:111
  838 tty1     Ss+    0:00 /sbin/agetty --noclear tty1 linux
  921 ?        Ss     0:00 dhclient -v -pf /run/dhclient.wlan0.pid -lf /var/lib/dhcp/dhclient.wlan0.leases wlan0
 1068 ?        Ss     0:00 sshd: pi [priv]
 1074 ?        R      0:00 sshd: pi@pts/0
 1076 pts/0    Ss     0:00 -bash
 1089 pts/0    S      0:00 su
 1094 pts/0    S      0:00 bash
:


... and they've stayed up. I set ArtCap looping and took 1439 1min images on the low software PC on the same switch over 27hours. I killed that connection and then connected to the same Pi from my scope PC, 4 switches away and downloaded 364 4min images in the following 25h and its still running as I write this. Although the managed switch shows no errors on the Ethernet connection to the Pi itself, it has ridden through DHCP lease renewals that killed the PuTTY connection but the images just keep coming rock steady. Mean time the two daughter processes are still there, with the artemishscervice process having clocked up 186 mins CPU - still the same process:
root@raspberrypi:/var/log# ps -ax | grep atik
  497 ?        R      1:39 /usr/lib/atikairservice
29464 pts/0    S+     0:00 grep atik
root@raspberrypi:/var/log# ps -ax | grep arte
  496 ?        Sl   186:18 /usr/lib/artemishscservice
29466 pts/0    S+     0:00 grep arte
root@raspberrypi:/var/log#


Also no memory growth (VIRT stuck in the band 91064 - 102872, %MEM 3.0 and same CPU fraction 5.5 to 6%). Dead impressed. Hope this helps iron out the last few bugs in the setup and service wrapper and especially the UI on the App. The USB comms and network link to the PC look to be solid when they get going properly.

Gerry

Aiming to post more on interfaces and wpa_supplicant files separately when I have more time as image users wanting to avoid too much Rasp Pi detail may want these and they are a little tricky especially matching to router setup.


razor

  • Newbie
  • *
  • Posts: 5
Re: AtikAirService & ArtemisHScService crashes
« Reply #6 on: November 23, 2016, 05:06:02 AM »
Great investigation, which reminded me that I did have to edit my network interfaces file to configure it with a manual entry as the automated confirmation resulted in atik air never being on the network as you described.

I am on wifi, curious about your testing where it stayed up solid, was this on a Ethernet connection?

Not that it should matter, I am on wifi with a static IP on a Pi 3.

And yes, this is one amazing piece of software. It will make me loyal to Atik products when thinking about other cams. Thank you Chris!
« Last Edit: November 23, 2016, 05:10:22 AM by razor »

ivygda

  • Newbie
  • *
  • Posts: 2
Re: AtikAirService & ArtemisHScService crashes
« Reply #7 on: November 23, 2016, 10:20:56 PM »
Thanks Razor,

All Ethernet for now I'm afraid. Havent yet sorted out a good wifi setup around the scope.

To get the Ethernet really reliable I had to throw out several cables and one seemingly ok switch. The config for this test has four netgear prosafe switches (lightweight management i/f but really useful for debugging problems and only few quid more than basic switch) 2xGS108Ev2, 1xGS108Ev3 and finally a GS108PEv2 (PoE) to the Rasp Pi using one of its non PoE ports.  Found these switches to reliably not drop packets - they have more buffer than basic switch. Had to swap out a high street purchased GS605 that I thought might be the same h/w but no management interface but it drops packets in careful testing. Turns out has smaller buffer - surprised this matters. Discovered all this tracing reliability for writing scope data direct to Synology Diskstation indoors - not on C D O drive (O C D in alphabetical order :-)

Will see if I can get wifi test going - camera is in loft (hence all the switches along route to scope) and may have decent wifi

I started down the path of getting wifi around scope good - I have an Arduino Yun doing driving 4 power switches that I wanted to access via wifi. But had to pause when I found the wifi from the house too weak. Had previously put an old wifi access point in wooden shed warm room but signal dropped to nothing when I lined it with modern double layer aluminised bubble insulation. I thought the gaps and slit window would leak enough RF but seemingly not :-(

FYI when I tried putting the in-house wifi AP near window to get signal to the scope it started misbehaving, regularly dropping stuff every few seconds like clockwork. Read around and saw that there are reports of problems with 5G wifi interacting with radar and apparently to get licence the AP has to drop out if sees radar. Cant be sure this is happening but AP reliability in window location came back when I switched off 5G band.

Have busy spell so may not be able to post wifi results for couple of days. Also want to suss out detailed behaviour of AtikAir app (any dependencies)

Gerry

astro_1

  • Newbie
  • *
  • Posts: 13
Re: AtikAirService & ArtemisHScService crashes
« Reply #8 on: November 28, 2016, 04:13:21 PM »
Have you tried or looked at my configuration .pdf I put together for making the Pi3 a wireless access point?

http://forums.atik-cameras.com/index.php?topic=2779.15

Jeff

razor

  • Newbie
  • *
  • Posts: 5
Re: AtikAirService & ArtemisHScService crashes
« Reply #9 on: December 02, 2016, 03:11:40 PM »
Restricting the PI to 2.4Ghz would be an interesting test.

Are you not getting any connection drops at all running the PI as an AP?