"dl_main: Assertion 'main_map != NULL' failed!" when executing mipsel binary

[spidermana]

Hi, I encountered some difficulties recently. I sincerely ask for a help.

These days, I am trying to execute a binary compiled for mips, so I write a quite simple program below:

int main(){
	int a;
	a=32;
}

and then compile it by command:mipsel-linux-gnu-gcc ./tests/hello.c -o ./tests/hello-mipsel.

After that, I run the binary through ./usercorn run ./tests/hello_mipsel

But it doesn't work and throw an assertion error.:
Inconsistency detected by ld.so: rtld.c: 983: dl_main: Assertion 'main_map != NULL' failed!

Details for trace:

0xc23c80: lb $v0, ($s6)                                      |                 | R c338a4
0xc23c84: bnez $v0, 0xc23bcc                                 |                
0xc23c88: move $at, $at                                      |                
0xc23c8c: lw $a0, 0x270($fp)                                 | a0 = 0x00000002 | R bfffe980
R 0x00c338a4: 00                                             [.                   ] R
R 0xbfffe980: 02000000                                       [....                ] R
0xc23c90: addiu $a1, $fp, 0x20                               | a1 = 0xbfffe730
0xc23c94: move $a2, $s5                                      | a2 = 0x0000000a
0xc23c98: addiu $v0, $zero, 0x1032                           | v0 = 0x00001032
Inconsistency detected by ld.so: rtld.c: 983: dl_main: Assertion `main_map != NULL' failed!
[note : Actually you can see a fault here!!!!]
0xc23c9c: syscall                                            | v0 = 0x0000005c
writev(2, 0xbfffe730, 0xa) = 0x5c
0xc23ca0: move $sp, $fp                                      | sp = 0xbfffe710
0xc23ca4: lw $ra, 0x26c($sp)                                 | ra = 0x00c241ec | R bfffe97c
0xc23ca8: lw $fp, 0x268($sp)                                 | s8 = 0xbfffe9d8 | R bfffe978
0xc23cac: lw $s7, 0x264($sp)                                 | s7 = 0x00c31000 | R bfffe974
0xc23cb0: lw $s6, 0x260($sp)                                 | s6 = 0x00c44ef0 | R bfffe970
0xc23cb4: lw $s5, 0x25c($sp)                                 | s5 = 0x00400034 | R bfffe96c
0xc23cb8: lw $s4, 0x258($sp)                                 | s4 = 0x00c452c8 | R bfffe968
0xc23cbc: lw $s3, 0x254($sp)                                 | s3 = 0x00c2b810 | R bfffe964
0xc23cc0: lw $s2, 0x250($sp)                                 | s2 = 0x00c2ea30 | R bfffe960
0xc23cc4: lw $s1, 0x24c($sp)                                 | s1 = 0x00000000 | R bfffe95c
0xc23cc8: lw $s0, 0x248($sp)                                 |                 | R bfffe958
0xc23ccc: jr $ra                                             |                
0xc23cd0: addiu $sp, $sp, 0x270                              | sp = 0xbfffe980
0xc241ec: lw $ra, 0x24($sp)                                  | ra = 0x00c2bdfc | R bfffe9a4
R 0xbfffe958: 00000000 00000000 30eac200 10b8c200 c852c400   [........0........R..] R
   0xbfffe96c: 34004000 f04ec400 0010c300 d8e9ffbf ec41c200   [4.@..N...........A..]
0xc241f0: jr $ra                                             |                
0xc241f4: addiu $sp, $sp, 0x28                               | sp = 0xbfffe9a8
0xc2bdfc: lw $gp, 0x20($sp)                                  |                 | R bfffe9c8
R 0xbfffe9a4: fcbdc200                                       [....                ] R
0xc2be00: lw $t9, -0x7f8c($gp)                               | t9 = 0x00c2e0a0 | R c45074
0xc2be04: bal 0xc2e0a0                                       | ra = 0x00c2be0c
0xc2be08: addiu $a0, $zero, 0x7f                             | a0 = 0x0000007f
0xc2e0a0: lui $gp, 2                                         | gp = 0x00020000
R 0xbfffe9c8: 00d0c400                                       [....                ] R
R 0x00c45074: a0e0c200                                       [....                ] R
0xc2e0a4: addiu $gp, $gp, -0x10a0                            | gp = 0x0001ef60
0xc2e0a8: addu $gp, $gp, $t9                                 | gp = 0x00c4d000
0xc2e0ac: lw $a2, -0x7ea8($gp)                               | a2 = 0x00c45bd8 | R c45158
0xc2e0b0: move $a1, $a0                                      | a1 = 0x0000007f
0xc2e0b4: move $a0, $a1                                      |                
0xc2e0b8: addiu $v0, $zero, 0x1096                           | v0 = 0x00001096
exit_group(127)
0xc2e0bc: syscall                                            | v0 = 0x00000000
R 0x00c45158: d85bc400                                       [.[..                ] R

Later, I search the source code of glibc/elf/rtld.c and glibc/elf/dl-object.c

//from rtld.c 
	main_map = _dl_new_object ((char *) "", "", lt_executable, NULL,
                                 __RTLD_OPENEXEC, LM_ID_BASE);
    assert (main_map != NULL); 

//from dl-object.c
  new = (struct link_map *) calloc (sizeof (*new) + audit_space
                                    + sizeof (struct link_map *)
                                    + sizeof (*newname) + libname_len, 1);
  if (new == NULL)
    return NULL;

Function calloc may return a NULL point. Unfortunately, I can't figure out why this happens. Is this error relevant to the usercorn emulation of 32-bit MIPS?

Besides, I also try to use qemu-mipsel to run the hello_mipsel and it can work successfully. But there is something wrong when using usercorn to execute this binary.

I am a newbie at this and therefore I need your help.
Any help or remark (or question) would be appreciated.
Thanks.

Details for the binary:

ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           MIPS R3000
  Version:                           0x1
  Entry point address:               0x4005a0
  Start of program headers:          52 (bytes into file)
  Start of section headers:          6660 (bytes into file)
  Flags:                             0x10001007, noreorder, pic, cpic, o32, mips2
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         11
  Size of section headers:           40 (bytes)
  Number of section headers:         33
  Section header string table index: 30

[lunixbochs]

Can you build the failing binary statically and upload it so I can run it?

[spidermana]

Thanks for your prompt reply!
And I upload source code, static-binary and dynamic-binary with ld.so and libc.so.

binary_for_mipsel.zip

Obviously I can't run the static binary, but the error is different with dynamic one.

Some instructions of static binary may trigger unmapped memory write

$ ./usercorn run ./tests/hello_mipsel_static 
invalid write: @0xffff9010, 0x4 = 0x3
----------------------------------------
Error: Invalid memory write (UC_ERR_WRITE_UNMAPPED)

In addition, I provide other information for you.

Trace for static binary:

getenv+0x168
0x40ac74: beqz $v0, 0x40ad14                                 |                
R 0xbfffed28: 78914800 78914800 04000000 28000000 00000000   [x.H.x.H.....(.......] R
   0xbfffed3c: 00004a00 202c4a00                              [..J. ,J.            ]
0x40ac78: lw $gp, 0x18($fp)                                  |                 | R bfffed60

__libc_message+0x6c
0x40ad14: lui $a0, 0x47                                      | a0 = 0x00470000
R 0xbfffed60: 40904a00                                       [@.J.                ] R
0x40ad18: addiu $a1, $zero, 0x882                            | a1 = 0x00000882
0x40ad1c: addiu $a0, $a0, 0x62a8                             | a0 = 0x004762a8
0x40ad20: addiu $v0, $zero, 0xfa5                            | v0 = 0x00000fa5
0x40ad24: syscall                                            | v0 = 0x00000003 | R 4762a8
open("/dev/tty", 2178, 0x4762a8) = 0x3

__libc_message+0x118
0x40ad28: beqz $a3, 0x40ae64                                 |                
R 0x004762a8: 2f646576 2f747479 00000000                     [/dev/tty....        ] R
0x40ad2c: move $s7, $v0                                      | s7 = 0x00000003

__libc_message+0x120
0x40ad30: lw $v0, -0x7654($gp)                               | v0 = 0xffff9010 | R 4a19ec
R 0x004a19ec: 00000000                                       [....                ] R
0x40ad34: rdhwr $v1, $29                                     | v1 = 0x00000000
0x40ad38: addu $v0, $v0, $v1                                 |                
0x40ad3c: b 0x40ac88                                         |                
invalid write: @0xffff9010, 0x4 = 0x3
Invalid memory write (UC_ERR_WRITE_UNMAPPED)
[pc]
0x40ad40: sw $s7, ($v0)
0x40ad44: lb $v1, 1($v0)
0x40ad48: bne $v1, $s2, 0x40acb4
0x40ad4c: move $at, $at
0x40ad50: bne $s4, $s0, 0x40acd4
0x40ad54: move $at, $at
0x40ad58: lb $v1, 1($s1)
0x40ad5c: bne $v1, $s2, 0x40acd4
0x40ad60: lw $t9, -0x7fc0($gp)
0x40ad64: lw $v0, 0x20($fp)
0x40ad68: addiu $s1, $s1, 2
0x40ad6c: addiu $s6, $s3, 1
0x40ad70: addiu $v1, $v0, 4
0x40ad74: lw $a1, ($v0)
0x40ad78: sw $v1, 0x20($fp)
0x40ad7c: move $a0, $a1
[memory map]
  0x400000-0x48e000 r-x [exe] ./tests/hello_mipsel_static
  0x49d000-0x4a3000 rw- [exe] ./tests/hello_mipsel_static(0x8d000)
  0x4a3000-0x4a4000 rw- [brk]
  0xbf800000-0xc0000000 rwx [stack]
[registers]
s0: 0x489178
s3: 0x28
s5: 0x4a0000
s7: 0x3
a1: 0x882
at: 0xfffffff8
a3: 0x80808080
s6: 0x4a2c20
s4: 0x0
t0: 0xfefefeff
t8: 0x10
sp: 0xbfffed48
gp: 0x4a9040
t3: 0x4a0ee4
t5: 0x1
t2: 0x0
s2: 0x4
t7: 0x4a1aa4
t6: 0x4a0000
s8: 0xbfffed48
ra: 0x40ac74
a0: 0x4762a8
t9: 0x4192a0
s1: 0x489178
a2: 0x4762a8
v1: 0x0
v0: 0xffff9010
t1: 0x2e
t4: 0x49f6b8
[callstack]
  0x0
  0x400640
  0x400888 __libc_start_main+0x8
  0x400ce8 __libc_setup_tls+0x8
  0x40af60 __libc_fatal+0x4
  0x40ac18 __libc_message+0x8
  0x40ad40 __libc_message+0x130
0x40ad40: sw $s7, ($v0)                                      |                 | W ffff9010
W 0xffff9010: 03000000                                       [....                ] W
----------------------------------------
Error: Invalid memory write (UC_ERR_WRITE_UNMAPPED)

version of mipsel-linux-gnu-gcc:

$ mipsel-linux-gnu-gcc -v
Using built-in specs.
COLLECT_GCC=mipsel-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/mipsel-linux-gnu/5/lto-wrapper
Target: mipsel-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.9' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libsanitizer --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-mipsel-cross/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-mipsel-cross --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-mipsel-cross --with-arch-directory=mips --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libgcj --enable-multiarch --disable-werror --enable-multilib --with-arch-32=mips2 --with-tune-32=mips32r2 --with-fp-32=xx --enable-targets=all --with-arch-64=mips3 --with-tune-64=mips64r2 --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=mipsel-linux-gnu --program-prefix=mipsel-linux-gnu- --includedir=/usr/mipsel-linux-gnu/include
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) 

[lunixbochs]

Sorry I'm coming back to this so late.

The static binary is failing in __libc_setup_tls

Here we can see when it calls rdhwr to read the thread-local storage register, it returns 0:
0x40ad34: rdhwr $v1, $29 | v1 = 0x00000000

There's probably an instruction or Linux syscall for MIPS to set the TLS register, maybe it's not implemented in usercorn or not working.

The dynamic binary returning 0 from calloc - can you find the call to calloc in the trace, see which syscalls it makes? Look for a brk or mmap.

[lunixbochs]

;; The TLS base pointer is accessed via "rdhwr $3, $29". No current
;; MIPS architecture defines this register, and no current
;; implementation provides it; instead, any OS which supports TLS is
;; expected to trap and emulate this instruction. rdhwr is part of the
;; MIPS 32r2 specification, but we use it on any architecture because
;; we expect it to be emulated. Use .set to force the assembler to
;; accept it.

This might be a unicorn limitation.

[spidermana]

Thanks for coming back and answering my questions.
It's been a while, so I rerun the dynamic binary with -strace.
mipsel_test_binary.zip

Details for strace:

spiderman@spiderman-virtual-machine:~/usercorn$ ./usercorn run -strace ./tests/test_mipsel 
brk(0x0) = 0x412000
mmap2(0x0, 0x1000, 3, 2050, 0, 0x0) = 0xc34000
Inconsistency detected by ld.so: rtld.c: 983: dl_main: Assertion `main_map != NULL' failed!
writev(2, 0xbfffe610, 0xa) = 0x5c
exit_group(127)

Therefore, mmap makes calloc returning 0.
Strangely, mmap2(0x0, 0x1000, 3, 2050, 0, 0x0) seems work well with the return value of 0xc34000.

I am sorry then, I didn't quite understand why the call to calloc matters?

Additionally, I contrast static-binary compiled by mipsel-linux-gnu-gcc in my machine
with usercorn/bins/mipsel.linux.elf.
Perhaps one difference is the version of ISA.

 spiderman@spiderman-virtual-machine:~/usercorn$ file ./bins/mipsel.linux.elf 
./bins/mipsel.linux.elf: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1 (SYSV), statically linked, not stripped
spiderman@spiderman-virtual-machine:~/usercorn$ file ./tests/test_mipsel_static 
./tests/test_mipsel_static: ELF 32-bit LSB executable, MIPS, MIPS-II version 1 (SYSV), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=0ba8f4b831b8bd28b11f6e69a1f85f247e5eda80, not stripped

I wonder if the limitation you mentioned can be interpreted in a way that only binary of MIPS 32r2 version can work well in usercorn.

[lunixbochs]

Can you run with -v so we can see the memory map layout on crash?

[spidermana]

The memory map specifically as follows:

spiderman@spiderman-virtual-machine:~/usercorn$ file ./tests/test_mipsel
./tests/test_mipsel: ELF 32-bit LSB executable, MIPS, MIPS-II version 1 (SYSV), dynamically linked, interpreter /lib/ld., for GNU/Linux 3.2.0, BuildID[sha1]=916234c4dc29ecdb318f5e33d2c3726cc323dedc, not stripped

spiderman@spiderman-virtual-machine:~/usercorn$ ./usercorn run -v -strace ./tests/test_mipsel
[entry @ 0xc11d00]
0xc11d00: move $t9, $ra
0xc11d04: bal 0xc11d0c
0xc11d08: move $at, $at
0xc11d0c: lui $gp, 4
0xc11d10: addiu $gp, $gp, -0x4d0c
0xc11d14: addu $gp, $gp, $ra
0xc11d18: move $ra, $t9
0xc11d1c: lw $a0, -0x7fe8($gp)
0xc11d20: sw $a0, -0x7ff0($gp)
0xc11d24: move $a0, $sp
0xc11d28: addiu $sp, $sp, -0x10
0xc11d2c: lw $t0, -0x7fe4($gp)
0xc11d30: addiu $t0, $t0, 0xd3c
0xc11d34: bltzal $t0, 0xc11d3c
0xc11d38: move $at, $at
0xc11d3c: subu $t0, $ra, $t0
[stack]
0xbfffed90: 00000000 00000000 00000000 00000000 00000000   [....................]
0xbfffeda4: 00000000 00000000 00000000                     [............        ]
[stack pointer]
0xbfffedb0: [skipped 0x64 null bytes]
[memory map]
  0x400000-0x401000 r-x [exe] ./tests/test_mipsel
  0x410000-0x412000 rw- [exe] ./tests/test_mipsel
  0xc11000-0xc34000 r-x [interp] /lib/ld.so.1
  0xc44000-0xc46000 rw- [interp] /lib/ld.so.1(0x23000)
  0xbf800000-0xc0000000 rwx [stack]
=====================================
==== Program output begins here. ====
=====================================
brk(0x0) = 0x412000
mmap2(0x0, 0x1000, 3, 2050, 0, 0x0) = 0xc34000
Inconsistency detected by ld.so: rtld.c: 983: dl_main: Assertion `main_map != NULL' failed!
writev(2, 0xbfffe610, 0xa) = 0x5c
exit_group(127)
[memory map]
  0x400000-0x401000 r-x [exe] ./tests/test_mipsel
  0x410000-0x412000 rw- [exe] ./tests/test_mipsel
  0xc11000-0xc34000 r-x [interp] /lib/ld.so.1
  0xc44000-0xc46000 rw- [interp] /lib/ld.so.1(0x23000)
  0xbf800000-0xc0000000 rwx [stack]
[registers]
ra: 0xc23e1c
a2: 0xa
s4: 0xbfffe81a
t0: 0xfefefeff
t1: 0x0
a0: 0x2
t3: 0xffffffff
t4: 0x20
a1: 0xbfffe610
s0: 0x0
s2: 0x0
s7: 0xbfffe810
t2: 0xa0000000
t5: 0xbfffe898
s1: 0xbfffe8a4
s6: 0xc338a4
s8: 0xbfffe5f0
v0: 0x1032
v1: 0xc338a4
sp: 0xbfffe5d8
a3: 0x80808080
s3: 0xbfffe610
s5: 0xa
t9: 0xc2e720
[callstack]
  0x0
  0x0
  0x0
  0x0

[lunixbochs]

That mmap seems bad, like it was mapped into the middle of ld.so!

mmap2(0x0, 0x1000, 3, 2050, 0, 0x0) = 0xc34000

  0xc11000-0xc34000 r-x [interp] /lib/ld.so.1
  0xc44000-0xc46000 rw- [interp] /lib/ld.so.1(0x23000)

[spidermana]

I have noticed that now!
Does usercorn lead to that ill-mmap? Would you fix it?
Or Could you offer a solution?

[spidermana]

I notice that now!
Does usercorn lead to that ill-mmap? Would you fix it?
Or Could you offer a solution?

[lunixbochs]

Can you run with the repl, e.g. ./usercorn run -repl and run the maps command to see what the mappings are before the target runs?

[spidermana]

spiderman@spiderman-virtual-machine:~/usercorn$ ./usercorn run -repl ./tests/test_mipsel
0xc11d00> maps
Memory map:
0x400000-0x401000 r-x [exe] ./tests/test_mipsel
0x410000-0x412000 rw- [exe] ./tests/test_mipsel
0xc11000-0xc34000 r-x [interp] /lib/ld.so.1
0xc44000-0xc46000 rw- [interp] /lib/ld.so.1(0x23000)
0xbf800000-0xc0000000 rwx [stack]
0xc11d00>  

[lunixbochs]

Oh, I bet this is related to thread local storage.

0x40ad34: rdhwr $v1, $29 from the static trace reads the TLS register and it comes back as 0
check the other trace for a similar instruction.

[lunixbochs]

See here: unicorn-engine/unicorn#407