Author Topic: Pipecom conversion to BAS only (MAJOR UPDATE!)  (Read 2704 times)

0 Members and 1 Guest are viewing this topic.

This topic contains a post which is marked as Best Answer or Most Recent Update. Press here if you would like to see it.

Offline SpriggsySpriggs

  • QB64 Developer
  • Forum Resident
  • Posts: 1090
  • If you're API and you know it clap your hands
    • My GitHub
Pipecom conversion to BAS only (MAJOR UPDATE!)
« on: February 10, 2021, 11:02:46 PM »
This is not the post you are looking for. CLICK the "Best Answer" post above ^^^!
« Last Edit: February 12, 2021, 09:16:53 PM by SpriggsySpriggs »
If you're API and you know it clap your hands
My GitHub

Offline NOVARSEG

  • Forum Resident
  • Posts: 510
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #1 on: February 11, 2021, 12:01:50 AM »
there is the readfile API!

 Function ReadFile% (ByVal hFile As Long, Byval lpBuffer As _Offset, Byval nNumberOfBytesToRead As Long, Byval lpNumberOfBytesRead As _Offset, Byval lpOverlapped As _Offset)


does pipe need writefile too?

in kernel 32?

where is kernel32 file?

« Last Edit: February 11, 2021, 12:04:01 AM by NOVARSEG »

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3451
    • Steve’s QB64 Archive Forum
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #2 on: February 11, 2021, 12:03:23 AM »
@SpriggsySpriggs — what you’re running into is Data Alignment/Packing.  Here’s a pretty good write up on what’s going on, down at Chapter 17: https://www.viva64.com/en/a/0004/#ID1C7DC5438C



Processors work more efficiently when they deal with data aligned properly. As a rule the 32-bit data item must be aligned at the border multiple of 4 bytes, and the 64-bit item at the border multiple of 8 bytes. An attempt to work with unaligned data on IA-64 (Itanium) processors will cause an exception as shown in the following example,.

#pragma pack (1) // Also set by key /Zp in MSVC
struct AlignSample {
  unsigned size;
  void *pointer;
} object;
void foo(void *p) {
  object.pointer = p; // Alignment fault
}
If you have to work with unaligned data on Itanium you should tell this to the compiler. For example, you may use a special macro UNALIGNED:

#pragma pack (1) // Also set by key /Zp in MSVC
struct AlignSample {
  unsigned size;
  void *pointer;
} object;
void foo(void *p) {
  *(UNALIGNED void *)&object.pointer = p; //Very slow
}
This solution is not efficient, because the access to the unaligned data will be several times slower. A better result may be achieved if you arrange up to 32-bit, 16-bit and 8-bit items in 64-bit data items.

On the x64 architecture during the access to unaligned data, an exception does not occur, but you should avoid them also. Firstly, because of the essential slowdown of the access to this data, and secondly, because of a high probability of porting the program on the IA-64 platform in the future.



Basically, you align your data on 8-byte breakpoints, rather than 4. 
The only point where I get lost in these conversions is how we determine those breakpoints.

dx AS LONG
dy AS LONG
dwSize AS LONG

Now, with the above, would we have EVERY long padded to 8 bytes?  Or would the first two variables (dx and dy) be counted together as 8-bytes, and fit the breakpoint?  Or do we padd the first and join the last two??

I *still* haven’t sat down and sorted out where the BLEEP we’re required to have those 8-byte breakpoints in data types.  All I can do is sit down in front of a keyboard, and furiously attempt to change first one and then the other, until I finally find a combo that works.
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline SpriggsySpriggs

  • QB64 Developer
  • Forum Resident
  • Posts: 1090
  • If you're API and you know it clap your hands
    • My GitHub
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #3 on: February 11, 2021, 12:04:12 AM »
@NOVARSEG ,

Please do not pursue the ReadFile API function. If you do, make a new thread.
If you're API and you know it clap your hands
My GitHub

Offline SpriggsySpriggs

  • QB64 Developer
  • Forum Resident
  • Posts: 1090
  • If you're API and you know it clap your hands
    • My GitHub
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #4 on: February 11, 2021, 12:07:18 AM »
@SMcNeill
@SpriggsySpriggs — what you’re running into is Data Alignment/Packing

I *still* haven’t sat down and sorted out where the BLEEP we’re required to have those 8-byte breakpoints in data types.  All I can do is sit down in front of a keyboard, and furiously attempt to change first one and then the other, until I finally find a combo that works.

Yep! I am aware of the alignment thing in Windows and I have struggled with it many times. It pisses me off to no end. The fact I'm only 8 bytes short in STARTUPINFO pisses me off even more.
If you're API and you know it clap your hands
My GitHub

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3451
    • Steve’s QB64 Archive Forum
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #5 on: February 11, 2021, 12:09:21 AM »
Cb as LONG
padding as LONG
.....


AND 4 more bytes padding near the end, would be my guess then.
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline SMcNeill

  • QB64 Developer
  • Forum Resident
  • Posts: 3451
    • Steve’s QB64 Archive Forum
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #6 on: February 11, 2021, 12:13:11 AM »
    Type STARTUPINFO
        cb As Long — 4 bytes + 4 bytes padding

        lpReserved As _Offset —8
        lpDesktop As _Offset — 8
        lpTitle As _Offset — 8
        dwX As Long +     dwY As Long — 8
        dwXSize As Long +   dwYSize As Long — 8
        dwXCountChars As Long + dwYCountChars As Long
        dwFillAttribute As Long +  dwFlags As Long

        wShowWindow As Integer + cbReserved2 As Integer = 4...  Probably need 4 padding

        lpReserved2 As _Offset
        hStdInput As Long +  hStdOutput As Long


        hStdError As Long + 4 padding


    End Type

I’d take a guess based off the above, since you’re only looking for 8-bytes total.
« Last Edit: February 11, 2021, 12:14:15 AM by SMcNeill »
https://github.com/SteveMcNeill/Steve64 — A github collection of all things Steve!

Offline NOVARSEG

  • Forum Resident
  • Posts: 510
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #7 on: February 11, 2021, 12:14:09 AM »
OK from my experience in assembler, I have encountered alignment problems as well.  The most hideous alignment problem is when an assembler instruction is say 32 bits and you try to load 16 bit data into the instruction.

The same thing happens from 32 bit to 64 bit

Offline NOVARSEG

  • Forum Resident
  • Posts: 510
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #8 on: February 11, 2021, 12:43:53 AM »
12 longs = 48 bytes
2 integers = 4 bytes
 total = 52 bytes

52 / 8 = 6.5

52  / 4 =  13

change integers to longs
wild  guess

« Last Edit: February 11, 2021, 12:48:04 AM by NOVARSEG »

Offline SpriggsySpriggs

  • QB64 Developer
  • Forum Resident
  • Posts: 1090
  • If you're API and you know it clap your hands
    • My GitHub
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #9 on: February 11, 2021, 01:12:30 AM »
Still no good on Windows x64. However, the below code works fine in Linux:

Code: QB64: [Select]
  1.  
  2. Print pipecom("ls")
  3.  
  4. Function pipecom$ (cmd As String)
  5.         Function popen%& (cmd As String, readtype As String)
  6.         Function feof& (ByVal stream As _Offset)
  7.         Function fgets$ (str As String, Byval n As Long, Byval stream As _Offset)
  8.         Function pclose& (ByVal stream As _Offset)
  9.     End Declare
  10.  
  11.     Dim As String pipecom_buffer
  12.     Dim As _Offset stream
  13.  
  14.     Dim buffer As String * 4096
  15.     stream = popen(cmd, "r")
  16.     If stream Then
  17.         While feof(stream) = 0
  18.             If fgets(buffer, 4096, stream) <> "" And feof(stream) = 0 Then
  19.                 pipecom_buffer = pipecom_buffer + Mid$(buffer, 1, InStr(buffer, Chr$(0)))
  20.             End If
  21.         Wend
  22.         Dim a As Long
  23.         a = pclose(stream)
  24.     End If
  25.     pipecom = pipecom_buffer

And this code works in 32 bit Windows:
Code: QB64: [Select]
  1.  
  2. Print pipecom("dir")
  3.  
  4. Function pipecom$ (cmd As String)
  5.     Type SECURITY_ATTRIBUTES
  6.         nLength As Long
  7.         lpSecurityDescriptor As _Offset
  8.         bInheritHandle As Long
  9.     End Type
  10.  
  11.     Type STARTUPINFO
  12.         cb As Long
  13.         lpReserved As _Offset
  14.         lpDesktop As _Offset
  15.         lpTitle As _Offset
  16.         dwX As Long
  17.         dwY As Long
  18.         dwXSize As Long
  19.         dwYSize As Long
  20.         dwXCountChars As Long
  21.         dwYCountChars As Long
  22.         dwFillAttribute As Long
  23.         dwFlags As Long
  24.         wShowWindow As Integer
  25.         cbReserved2 As Integer
  26.         lpReserved2 As _Offset
  27.         hStdInput As Long
  28.         hStdOutput As Long
  29.         hStdError As Long
  30.     End Type
  31.  
  32.     Type PROCESS_INFORMATION
  33.         hProcess As Long
  34.         hThread As Long
  35.         dwProcessId As Long
  36.         dwThreadId As Long
  37.     End Type
  38.  
  39.     Const TRUE = 1
  40.     Const FALSE = 0
  41.     Const STARTF_USESTDHANDLES = &H00000100
  42.     Const CREATE_NO_WINDOW = &H8000000
  43.  
  44.     Declare Dynamic Library "Kernel32"
  45.         Function CreatePipe% (ByVal hReadPipe As _Offset, Byval hWritePipe As _Offset, Byval lpPipeAttributes As _Offset, Byval nSize As Long)
  46.         Function CreateProcess% Alias CreateProcessA (ByVal lpApplicationName As _Offset, Byval lpCommandLine As _Offset, Byval lpProcessAttributes As _Offset, Byval lpThreadAttributes As _Offset, Byval bInheritHandles As Integer, Byval dwCreationFlags As Long, Byval lpEnvironment As _Offset, Byval lpCurrentDirectory As _Offset, Byval lpStartupInfor As _Offset, Byval lpProcessInformation As _Offset)
  47.         Function CloseHandle% (ByVal hObject As Long)
  48.         Function ReadFile% (ByVal hFile As Long, Byval lpBuffer As _Offset, Byval nNumberOfBytesToRead As Long, Byval lpNumberOfBytesRead As _Offset, Byval lpOverlapped As _Offset)
  49.         Function GetStdHandle& (ByVal nStdHandle As Long)
  50.         Function GetLastError& ()
  51.     End Declare
  52.  
  53.     Dim As String pipecom_buffer
  54.     Dim As Integer ok: ok = TRUE
  55.     Dim As Long hStdOutPipeRead
  56.     Dim As Long hStdOutPipeWrite
  57.     Dim As SECURITY_ATTRIBUTES sa: sa.nLength = Len(sa): sa.lpSecurityDescriptor = 0: sa.bInheritHandle = TRUE
  58.  
  59.     If CreatePipe(_Offset(hStdOutPipeRead), _Offset(hStdOutPipeWrite), _Offset(sa), 0) = FALSE Then
  60.         pipecom = "oops1"
  61.         Print GetLastError
  62.         Exit Function
  63.     End If
  64.  
  65.     Dim As STARTUPINFO si
  66.     si.cb = Len(si)
  67.     si.dwFlags = STARTF_USESTDHANDLES
  68.     si.hStdError = 0
  69.     si.hStdOutput = hStdOutPipeWrite
  70.     si.hStdInput = 0
  71.     Dim As PROCESS_INFORMATION pi
  72.     Dim As _Offset lpApplicationName
  73.     Dim As String fullcmd: fullcmd = "cmd /c " + cmd + Chr$(0)
  74.     Dim As String lpCommandLine: lpCommandLine = fullcmd
  75.     Dim As Long lpProcessAttributes
  76.     Dim As Long lpThreadAttributes
  77.     Dim As Integer bInheritHandles: bInheritHandles = TRUE
  78.     Dim As Long dwCreationFlags: dwCreationFlags = CREATE_NO_WINDOW
  79.     Dim As _Offset lpEnvironment
  80.     Dim As _Offset lpCurrentDirectory
  81.     ok = CreateProcess(lpApplicationName,_
  82.                        _Offset(lpCommandLine),_
  83.                        lpProcessAttributes,_
  84.                        lpThreadAttributes,_
  85.                        bInheritHandles,_
  86.                        dwCreationFlags,_
  87.                        lpEnvironment,_
  88.                        lpCurrentDirectory,_
  89.                        _Offset(si),_
  90.                        _Offset(pi))
  91.     If ok = FALSE Then
  92.         pipecom = "oops2"
  93.         Print GetLastError
  94.     End If
  95.  
  96.     ok = CloseHandle(hStdOutPipeWrite)
  97.  
  98.     Dim As String buf: buf = Space$(4096 + 1)
  99.     Dim As Long dwRead
  100.     While ReadFile(hStdOutPipeRead, _Offset(buf), 4096, _Offset(dwRead), 0) <> 0 And dwRead
  101.         Mid$(buf, InStr(buf, Chr$(13)), 1) = Chr$(0)
  102.         pipecom_buffer = pipecom_buffer + Mid$(buf, 1, dwRead)
  103.     Wend
  104.     Print GetLastError
  105.     ok = CloseHandle(hStdOutPipeRead)
  106.     pipecom = pipecom_buffer

Only one small hop to go before pipecom becomes only an $INCLUDE rather than a header
« Last Edit: February 11, 2021, 01:13:41 AM by SpriggsySpriggs »
If you're API and you know it clap your hands
My GitHub

Offline NOVARSEG

  • Forum Resident
  • Posts: 510
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #10 on: February 11, 2021, 01:23:21 AM »
noticed that all the variables that start with lp are as _offset.

104 bytes in  Type STARTUPINFO you say?

12 longs = 48 bytes
2 integers = 4 bytes
 total = 52 bytes

52 * 2 = 104 ???

change the integers to longs and the longs to 64 bit

another wild guess
« Last Edit: February 11, 2021, 01:25:54 AM by NOVARSEG »

Offline SpriggsySpriggs

  • QB64 Developer
  • Forum Resident
  • Posts: 1090
  • If you're API and you know it clap your hands
    • My GitHub
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #11 on: February 11, 2021, 08:52:53 AM »
@NOVARSEG This has already been tried.
If you're API and you know it clap your hands
My GitHub

Offline Petr

  • Forum Resident
  • Posts: 1598
  • The best code is the DNA of the hops.
Re: Conversion for pipecom to BAS only (Help, Petr!)
« Reply #12 on: February 11, 2021, 10:41:54 AM »
I try it, thank for your trust :)

Offline SpriggsySpriggs

  • QB64 Developer
  • Forum Resident
  • Posts: 1090
  • If you're API and you know it clap your hands
    • My GitHub
Pipecom conversion to BAS only (SOLVED!)
« Reply #13 on: February 11, 2021, 10:44:19 AM »
@Petr @SMcNeill @NOVARSEG @bplus @doppler

Stack Overflow got it fixed! Now pipecom works without a header! You can simply $INCLUDE the below code and it will work in Windows 32&64, Linux, and Mac!

Code: QB64: [Select]
  1.  
  2. Print pipecom("dir")
  3.  
  4. 'begin include
  5. Function pipecom$ (cmd As String)
  6.     $If WIN Then
  7.         Type SECURITY_ATTRIBUTES
  8.             nLength As Long
  9.             $If 64BIT Then
  10.                 padding As Long
  11.             $End If
  12.             lpSecurityDescriptor As _Offset
  13.             bInheritHandle As Long
  14.             $If 64BIT Then
  15.                 padding2 As Long
  16.             $End If
  17.         End Type
  18.  
  19.         Type STARTUPINFO
  20.             cb As Long
  21.             $If 64BIT Then
  22.                 padding As Long
  23.             $End If
  24.             lpReserved As _Offset
  25.             lpDesktop As _Offset
  26.             lpTitle As _Offset
  27.             dwX As Long
  28.             dwY As Long
  29.             dwXSize As Long
  30.             dwYSize As Long
  31.             dwXCountChars As Long
  32.             dwYCountChars As Long
  33.             dwFillAttribute As Long
  34.             dwFlags As Long
  35.             wShowWindow As Integer
  36.             cbReserved2 As Integer
  37.             $If 64BIT Then
  38.                 padding2 As Long
  39.             $End If
  40.             lpReserved2 As _Offset
  41.             hStdInput As _Offset
  42.             hStdOutput As _Offset
  43.             hStdError As _Offset
  44.         End Type
  45.  
  46.         Type PROCESS_INFORMATION
  47.             hProcess As _Offset
  48.             hThread As _Offset
  49.             dwProcessId As Long
  50.             $If 64BIT Then
  51.                 padding2 As Long
  52.             $End If
  53.         End Type
  54.  
  55.         Const STARTF_USESTDHANDLES = &H00000100
  56.         Const CREATE_NO_WINDOW = &H8000000
  57.  
  58.         Declare Dynamic Library "Kernel32"
  59.             Function CreatePipe% (ByVal hReadPipe As _Offset, Byval hWritePipe As _Offset, Byval lpPipeAttributes As _Offset, Byval nSize As Long)
  60.             Function CreateProcess% Alias CreateProcessA (ByVal lpApplicationName As _Offset, Byval lpCommandLine As _Offset, Byval lpProcessAttributes As _Offset, Byval lpThreadAttributes As _Offset, Byval bInheritHandles As Integer, Byval dwCreationFlags As Long, Byval lpEnvironment As _Offset, Byval lpCurrentDirectory As _Offset, Byval lpStartupInfor As _Offset, Byval lpProcessInformation As _Offset)
  61.             Function CloseHandle% (ByVal hObject As Long)
  62.             Function ReadFile% (ByVal hFile As Long, Byval lpBuffer As _Offset, Byval nNumberOfBytesToRead As Long, Byval lpNumberOfBytesRead As _Offset, Byval lpOverlapped As _Offset)
  63.         End Declare
  64.  
  65.         Dim As String pipecom_buffer
  66.         Dim As Integer ok: ok = 1
  67.         Dim As Long hStdOutPipeRead
  68.         Dim As Long hStdOutPipeWrite
  69.         Dim As SECURITY_ATTRIBUTES sa: sa.nLength = Len(sa): sa.lpSecurityDescriptor = 0: sa.bInheritHandle = 1
  70.  
  71.         If CreatePipe(_Offset(hStdOutPipeRead), _Offset(hStdOutPipeWrite), _Offset(sa), 0) = 0 Then
  72.             pipecom = ""
  73.             Exit Function
  74.         End If
  75.  
  76.         Dim As STARTUPINFO si
  77.         si.cb = Len(si)
  78.         si.dwFlags = STARTF_USESTDHANDLES
  79.         si.hStdError = 0
  80.         si.hStdOutput = hStdOutPipeWrite
  81.         si.hStdInput = 0
  82.         Dim As PROCESS_INFORMATION pi
  83.         Dim As _Offset lpApplicationName
  84.         Dim As String fullcmd: fullcmd = "cmd /c " + cmd + " 2>&1" + Chr$(0)
  85.         Dim As String lpCommandLine: lpCommandLine = fullcmd
  86.         Dim As Long lpProcessAttributes
  87.         Dim As Long lpThreadAttributes
  88.         Dim As Integer bInheritHandles: bInheritHandles = 1
  89.         Dim As Long dwCreationFlags: dwCreationFlags = CREATE_NO_WINDOW
  90.         Dim As _Offset lpEnvironment
  91.         Dim As _Offset lpCurrentDirectory
  92.         ok = CreateProcess(lpApplicationName,_
  93.                            _Offset(lpCommandLine),_
  94.                            lpProcessAttributes,_
  95.                            lpThreadAttributes,_
  96.                            bInheritHandles,_
  97.                            dwCreationFlags,_
  98.                            lpEnvironment,_
  99.                            lpCurrentDirectory,_
  100.                            _Offset(si),_
  101.                            _Offset(pi))
  102.         If ok = 0 Then
  103.             pipecom = ""
  104.             Exit Function
  105.         End If
  106.  
  107.         ok = CloseHandle(hStdOutPipeWrite)
  108.  
  109.         Dim As String buf: buf = Space$(4096 + 1)
  110.         Dim As Long dwRead
  111.         While ReadFile(hStdOutPipeRead, _Offset(buf), 4096, _Offset(dwRead), 0) <> 0 And dwRead > 0
  112.             buf = Mid$(buf, 1, dwRead)
  113.             buf = String.Remove(buf, Chr$(13))
  114.             pipecom_buffer = pipecom_buffer + buf
  115.             buf = Space$(4096 + 1)
  116.         Wend
  117.         ok = CloseHandle(hStdOutPipeRead)
  118.         pipecom = pipecom_buffer
  119.     $Else
  120.         Function popen%& (cmd As String, readtype As String)
  121.         Function feof& (ByVal stream As _Offset)
  122.         Function fgets$ (str As String, Byval n As Long, Byval stream As _Offset)
  123.         Function pclose& (ByVal stream As _Offset)
  124.         End Declare
  125.  
  126.         Dim As String pipecom_buffer
  127.         Dim As _Offset stream
  128.  
  129.         Dim buffer As String * 4096
  130.         stream = popen(cmd+ " 2>&1", "r")
  131.         If stream Then
  132.         While feof(stream) = 0
  133.         If fgets(buffer, 4096, stream) <> "" And feof(stream) = 0 Then
  134.         pipecom_buffer = pipecom_buffer + Mid$(buffer, 1, InStr(buffer, Chr$(0)) - 1)
  135.         End If
  136.         Wend
  137.         Dim As Long a
  138.         a = pclose(stream)
  139.         End If
  140.         pipecom = pipecom_buffer
  141.     $End If
  142.  
  143. $If WIN Then
  144.     Function String.Remove$ (a As String, b As String)
  145.         Dim c As String
  146.         Dim j
  147.         Dim r$
  148.         c = ""
  149.         j = InStr(a, b)
  150.         If j > 0 Then
  151.             r$ = Left$(a, j - 1) + c + String.Remove(Right$(a, Len(a) - j + 1 - Len(b)), b)
  152.         Else
  153.             r$ = a
  154.         End If
  155.         String.Remove = r$
  156. 'end include
« Last Edit: February 11, 2021, 06:47:40 PM by SpriggsySpriggs »
If you're API and you know it clap your hands
My GitHub

Online bplus

  • Forum Resident
  • Posts: 7044
  • b = b + ...
Re: Pipecom conversion to BAS only (SOLVED!)
« Reply #14 on: February 11, 2021, 03:01:49 PM »
@SpriggsySpriggs

Nice! but "di" errors in Windows 10 I think you meant "dir"?

Anyway I tested with my pipecom call that I used in browser, it seems to have returned xtra chars at first char in line, possibly a line delimiter conflict? (see attched)