Anonymous pipe: wierdness

11 posts / 0 new

Last post

Thu, 2015-07-02 13:14

ChrisH

Offline

Last seen: 2 years 3 months ago

Joined: 2010-12-07 20:22

Anonymous pipe: wierdness

The following AmigaDOS script works as expected:

Echo "Hello One" >PIPE:
Echo "Hello Two" >PIPE:
 
Echo "1st PIPE: " NOLINE
Type PIPE:
Echo "2nd PIPE: " NOLINE
Type PIPE:

It outputs:

1st PIPE: Hello One 2nd PIPE: Hello Two

However, when I try to do a similar thing using compiled code, it does not do what I expect:

MODULE 'dos'
 
PROC main()
	DEF inp1:BPTR, out1:BPTR
	DEF inp2:BPTR, out2:BPTR
	DEF len, buffer[100]:STRING
 
	out1 := Open('PIPE:', MODE_OLDFILE)
	inp1 := Open('PIPE:', MODE_NEWFILE)
 
	out2 := Open('PIPE:', MODE_OLDFILE)
	inp2 := Open('PIPE:', MODE_NEWFILE)
 
	Write(inp1, 'Hello One', STRLEN)
	Write(inp2, 'Hello Two', STRLEN)
 
	IF inp1 THEN Close(inp1) ; inp1 := NIL
	IF inp2 THEN Close(inp2) ; inp2 := NIL
 
	len := Read(out1, buffer, 100) ; SetStr(buffer, len) ; Print('out1 => "\s"\n', buffer)
	len := Read(out2, buffer, 100) ; SetStr(buffer, len) ; Print('out2 => "\s"\n', buffer)
FINALLY
	PrintException()
 
	IF inp1 THEN Close(inp1)
	IF inp2 THEN Close(inp2)
	IF out1 THEN Close(out1)
	IF out2 THEN Close(out2)
ENDPROC

That's PortablE code, but it's close enough to C that you should be able to see basically what I'm doing. Unfortunately it output's the following before permanently hanging on the second Pipe:

out1 => "Hello OneHello Two"

I have tried the following variation of my code (calling Write() & Close() earlier), with no change/improvement:

MODULE 'dos'
 
PROC main()
	DEF inp1:BPTR, out1:BPTR
	DEF inp2:BPTR, out2:BPTR
	DEF len, buffer[100]:STRING
 
	out1 := Open('PIPE:', MODE_OLDFILE)
	inp1 := Open('PIPE:', MODE_NEWFILE)
	Write(inp1, 'Hello One', STRLEN)
	IF inp1 THEN Close(inp1) ; inp1 := NIL
 
	out2 := Open('PIPE:', MODE_OLDFILE)
	inp2 := Open('PIPE:', MODE_NEWFILE)
	Write(inp2, 'Hello Two', STRLEN)
	IF inp2 THEN Close(inp2) ; inp2 := NIL
 
	len := Read(out1, buffer, 100) ; SetStr(buffer, len) ; Print('out1 => "\s"\n', buffer)
	len := Read(out2, buffer, 100) ; SetStr(buffer, len) ; Print('out2 => "\s"\n', buffer)
FINALLY
	PrintException()
 
	IF inp1 THEN Close(inp1)
	IF inp2 THEN Close(inp2)
	IF out1 THEN Close(out1)
	IF out2 THEN Close(out2)
ENDPROC

Can someone please point out what I am doing wrong?

Thu, 2015-07-02 18:59

broadblues

Offline

Last seen: 2 years 9 months ago

Joined: 2012-05-02 21:48

I was slighly suprised your

I was slighly suprised your inital example from the shell worked. But verifying it showed that it does. But you src following is dosing something quite different. Consider the following, first a verification at the shell that confirms your expacted behaviour, then a short interactive python session.

New Shell process 8
8.AmigaOS4:> echo "foo one" >PIPE:
8.AmigaOS4:> echo "foo two" >PIPE:
8.AmigaOS4:> type PIPE:
foo one
8.AmigaOS4:> type PIPE:
foo two
8.AmigaOS4:> python
Python 2.5.6 (r256:88840, Oct  5 2014, 20:56:34) 
[GCC 4.2.4 (adtools build 20090118)] on ppc-amiga
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open("PIPE:","w")
>>> f.write("foo one")
>>> f.close()
>>> f = open("PIPE:","w")
>>> f.write("foo two")
>>> f.close()
>>> f = open("PIPE:","r")
>>> f.read()
'foo one'
>>> f.close()
>>> f = open("PIPE:","r")
>>> f.read()
'foo two'
>>> f.close()
>>>

So it can work programaticaly too. What might be wrong, opening the read end first? I think you need to open the write end of an anonymous pipe first. Interleaving your read and write opens may not be helping. For safe and predicatble usage in script or program like tis you should nearly always be using named pipes!

Thu, 2015-07-02 19:10

(Reply to #2) #3

broadblues

Offline

Last seen: 2 years 9 months ago

Joined: 2012-05-02 21:48

Further interactive testing

Further interactive testing strongly suggests that the Write pipe must be closed for the next read pipe to open the same pipe, if two anonymous pipes are created simulataneously by opening both ends they do not point to the same pipe. but in the process of testing I ended up with a number of hanging pipes and no way to ensure which pipe my opens were opening, thus messing up my system. So I'll say it again don't use anonymous pipes....

Thu, 2015-07-02 19:36

(Reply to #3) #4

tbreeden

Offline

Last seen: 4 years 8 months ago

Joined: 2010-12-09 03:10

I'll second that from

I'll second that from broadblues. I suspect there are always problems if all readers close on a pipe before that only writer closes. FWIW, here are some notes I made to myself a while back; they may not be correct after a number of os AOS 4 updates.

ON AmigaOS 4.0 > Apparently there are problems if the reader first closes and then the writer. Get hangs and illegal memory reference gurus. Seems it is possible, however, for the reader to close, another reader to open, then the writer close, and then the second reader close. AOS4 experiments as of 1/22/08 > if, when the reader is opened no writer is open, it blocks > if the reader stays open, multiple writers can open and write to it > if the reader is open and the last writer closes, the reader will return 0 on the first read and thereafter continue to return -1 until it is closed, irrespective of whether new writers open. > The reader must then be closed and re-opened to start again with a useful pipe. Most pending writer data will be then read, though some may be lost.

Tom

Thu, 2015-07-02 20:30

tbreeden

Offline

Last seen: 4 years 8 months ago

Joined: 2010-12-09 03:10

I think that the first MODULE

I think that the first MODULE does give the expected results:

out1 => "Hello OneHello Two"

Since it looks like your PortablE code is using the DOS library directly. ie, the first Read() of up to 100 bytes will just empty the current contents of the PIPE. With a LF included at the end of each Write() it might come out as:

out1 => "Hello One Hello Two"

To get the best speed, I ended up writing my own proc to DOS.Read() as much as possible, but allow LF delimited reads as well. (taking this opportunity to post Modula-2 code :) You may be able to use DOS FReadLine() instead. This, along with large sizes for /[Buffer Size] and /[Max Buffers] in the Open call, will give very fast PIPE transferring of data. os4depot game

(*-----------------------------------------------------------------------------------------------*)
 PROCEDURE ReadToLF(file:PipeFile; VAR Dest:GenStr; maxsize:INTEGER; PutChar:PutCharProc):BOOLEAN;
(*-----------------------------------------------------------------------------------------------*)
 
VAR inx       :INTEGER;
    LFfound   :BOOLEAN;
    ch        :CHAR;
 
BEGIN WITH file^ DO
 
inx := 0;
LFfound := FALSE;
 
LOOP
   IF TstBreak() THEN           (* not very useful currently *)
      pfReadCnt := 0;
      EXIT;
   END;
 
   IF pfNeedRead THEN
      pfReadCnt := ReadBuffered(file);
   END;
 
   IF pfReadCnt = -2 THEN   (* timed out *)
      IF pfRTOExceptOn THEN
         RAISE(PipeIOExceptSrc, ORD(PipeIOTimeOut), "PipeIOTimeOut");
      END;
      EXIT;
 
   ELSIF pfReadCnt > 0 THEN
      WHILE NOT LFfound AND NOT pfNeedRead DO
         ch := pfLineBuffer[pfBufPos];
         INC(pfBufPos);
         IF pfBufPos >= pfReadCnt THEN
            pfNeedRead := TRUE;
         END;
 
         IF ch # LF THEN
            PutChar(ch, Dest, inx, maxsize);
         ELSE
            LFfound := TRUE;
         END;
      END;
      IF LFfound THEN
        EXIT;
      END;
   END;
END;
 
RETURN LFfound;
 
END END ReadToLF;

Using the "PipeFile" type:

TYPE PipeFileRec = RECORD
                      pfFile         :FileHandle(*BPTR*);
                      pfBufPos       :INTEGER;
                      pfReadCnt      :INTEGER;
                      pfRTimMillis   :INTEGER;
                      pfFileName     :Str0.String80;
                      pfLineBuffer   :ARRAY[0..ReadBufSize] OF CHAR;
                      pfNeedRead     :BOOLEAN;
                      pfReadMode     :BOOLEAN;
                      pfRTOExceptOn  :BOOLEAN;  (* if TRUE, a read timeout causes an Exception *)
                      pfRTO          :BOOLEAN;  (* read timed out *)
                    END;

Sat, 2015-08-01 14:46

ChrisH

Offline

Last seen: 2 years 3 months ago

Joined: 2010-12-07 20:22

Re: Anonymous pipe: wierdness

Sorry for late reply. Too many posts to answer directly, so I'll just answer the main general points: My reason for wanting to use anonymous pipes is that it avoids the need to generate a pseudo-random filename that might occasionally conflict with an existing filename on the pipe (leading to buggy behaviour). I can open both ends of an anonymous pipe within a very short time period, and do so using a raised task priority, so that there is an insignificant chance of anything else (e.g. a script) stealing my anonymous pipe from me. My compiled code was NOT trying to exactly implement the shell script (which is why I only said "do a *similar* thing"). In fact the shell script was trying to do what my compiled code does, within the limitation of the shell, hence the differences. @broadblues

What might be wrong, opening the read end first? I think you need to open the write end of an anonymous pipe first.

I believe that I already tried that, without any improvement.

Further interactive testing strongly suggests that the Write pipe must be closed for the next read pipe to open the same pipe, if two anonymous pipes are created simulataneously by opening both ends they do not point to the same pipe.

That's a real shame, as I need both Write pipes to be simultaneously open for quite a long time. But even if I *could* get it to work, it seems like such behaviour MIGHT vary dramatically between different version of OS4, and even more likely between different Amiga-like flavours. So it seems that I *have* to use named pipes, despite my reservations. :( Luckily I managed to get the pseudo-random filenames reliably different, so hopefully only a low chance of a conflict...

Author of the PortablE programming language. I love using Amiga OS4.1 on my X1000 & Sam440 :-D

Sun, 2015-08-02 19:31

hypex

Offline

Last seen: 1 month 3 weeks ago

Joined: 2011-09-09 16:20

Re: Anonymous pipe: wierdness

Why do you write to your input file and read from your output file? ;-) Just looks backwards to me. Also, you open a pipe twice per once access. Once to read, another to write. So this would be a diffenece to the shell script. Have you tried opening only one pipe for both read and write access? You could also try using one pipe only. But to duplicate the script, you'd need to open a pipe, write to it and close. Twice for each write. And then do the same when reading it back out again. Also, your E code doesn't send line feeds that I can see, so also try and add one there. I think that may affect the bufferieng of the pipe, if it flushes on a line feed.

Mon, 2015-08-03 15:17

(Reply to #7) #8

thomas

Offline

Last seen: 16 hours 8 min ago

Joined: 2011-05-16 14:23

Re: Anonymous pipe: wierdness

Unless the maintainter of pipe-handler states otherwise, I would expect that such thing as an anonymous pipe does not exist. All references to PIPE: refer to the same one pipe. Just consider PIPE: a named pipe with a null name. At least this would explain your observations.

Tue, 2015-08-04 17:55

(Reply to #8) #9

ChrisH

Offline

Last seen: 2 years 3 months ago

Joined: 2010-12-07 20:22

Re: Anonymous pipe: wierdness

@hypex

Why do you write to your input file and read from your output file? ;-) Just looks backwards to me.

Hmmm, yes, you are right. I think I fixed that in the real test code, and somehow forgot to update my post.

Have you tried opening only one pipe for both read and write access?

That would be useless for my needs. I need both pipes operating simultaneously. I am not trying to duplicate the script (see my previous post). I sincerely hope that line feeds make no difference, because binary data gets sent through the pipes! @thomas

Unless the maintainter of pipe-handler states otherwise, I would expect that such thing as an anonymous pipe does not exist.

It was broadblues (I think) on this site that introduced Amiga-style anonymouse pipes to me. I hadn't come across them before, but they do seem to exist, at least in some form.

Author of the PortablE programming language. I love using Amiga OS4.1 on my X1000 & Sam440 :-D

Wed, 2015-08-05 20:37

(Reply to #9) #10

ChrisH

Offline

Last seen: 2 years 3 months ago

Joined: 2010-12-07 20:22

Re: Anonymous pipe: wierdness

@hypex

Why do you write to your input file and read from your output file? ;-) Just looks backwards to me.

Having double-checked it, my code is correct. It's just that the variable names for input & output might be backwards to what you may expect! e.g. "inp1" is opened using MODE_NEWFILE, and written to using Write(). "out1" is opened using MODE_OLDFILE, and read from using Read(). These names are from the perspective of the Pipe itself, rather than the user of the Pipe. P.S. This 30-minute editing limit is really annoying. Now I have an entirely incorrect part in my earlier reply, and I cannot remove it.

Author of the PortablE programming language. I love using Amiga OS4.1 on my X1000 & Sam440 :-D

Sat, 2015-08-22 16:30

(Reply to #10) #11

hypex

Offline

Last seen: 1 month 3 weeks ago

Joined: 2011-09-09 16:20

Re: Anonymous pipe: wierdness

@ChrisH

That would be useless for my needs. I need both pipes operating simultaneously.

My point was it looks complicated. There are four pipes in operation. That could be encumbering the behaviour.

I am not trying to duplicate the script (see my previous post).

Does that mean your script was trying to reproduce the hang problem? ;-)

I sincerely hope that line feeds make no difference

I brought it to mind because the Echo command would end up sending it unless you did a NOLINE. Have you tried adding "\n" to your written strings?

These names are from the perspective of the Pipe itself, rather than the user of the Pipe.

Okay I understand. I wonder if opening different pipe files has an affect on the pipe, but from your results it looks like it uses the same one. Regarding named pipes and creating an ID. I'm sure I saw a function for creating a UUID for this sort of purpose. But even if not we can look to the shell which uses the shell ID in scripts for this type of purpose. And in program code the obvious thing to do would be to take the task ID and use that to create an ID string you can use as a UUID.

P.S. This 30-minute editing limit is really annoying.

Yeah I recall it used to cut off when someone added a post. And that was annoying enough. :-)

Search form

User login