Vyxal, 17 bytes

2ʀ9↔µ∑;«+⅛«İṠḟbṅḢ

Try it Online!

This relies on a couple of very nice coincidences:

• Padding the output to length 13 would cost 4 bytes 13∆Z. One thing we could do to avoid this is, instead of mapping the input to the range [0, 6045], instead map it to some subset of [2^13, 2^14) or similar, resulting in 14-bit strings starting with a 1, from which we can remove the first character.
• Valid boards should contain at most 1 more X than O, which is equivalent to mapping x to 1, o to -1 and to 0, and checking if the sum is 0 (same amount) or 1 (one more x). This is equivalent to mapping o x to 012 and checking if the sum is 9 or 10.
• Now for the neat part: There are 8272 boards with sum at most 8, and 5365 boards with sum at least 11. This means that, if we construct a list of all the boards and sort it by their sums, the valid boards with sums 9 and 10 will be between indices 8272 and 14317, which is within the desired 14-bit range, so all we have to do is find the index of the input in that list, convert it to binary and remove the first character.

The caveat of all this is that it's quite slow, taking 30-60 seconds on my computer. Because it's constructing a list of every board, it takes a while, and for some reason ḟinding the input's index in that list takes 20+ seconds. As such, the above link will probably time out.

             ḟ    # Find the index of the input in
  9↔              # Combinations of length 9 of
2ʀ                # [0, 1, 2] 
    µ ;           # Sorted by
     ∑            # their sum
           İ      # Index into
       «+⅛«       # "o x"
            Ṡ     # And concatenate
              bṅ  # Convert the index to binary and concatenate
                Ḣ # And remove the first character

Perl 5 -n, 97 bytes

$i=$_;$n=0;y/123/ ox/==9&&(s/x/x/g- s/o/o/g)=~/0|^1/&&++$n&&/$i/&&printf"%013b\n",$n for 1e8..4e8

Try it online!

Very slow as it loops from 100 000 000 to 400 000 000 (base-10) throwing away all numbers containing any other digits than 1s, 2s and 3s by checking if y/123/ ox/ returns 9 for the number of chars replaced where it translates 1 2 3 into o x accordingly. Then where the count of xs is the same as or one more than the count of os the counter variable $n is increased by one since this is a valid input. When the now translated string of nine , o or xs equals the input string, it outputs the 13 digit binary representation of the current $n.

Perl 5, 142 bytes

sub{local($_,$n)=$"x9;do{(s/x/x/g- s/o/o/g)!~/0|^1/ or$f{$_}=sprintf"%013b",$n++}while s/([ o])x*$/($1eq$"?o:x).($"x(-1+length$&))/e;$f{+pop}}

Try it online!

Second answer. This function gives each of the 6045 valid input strings a unique identifier with 13 bits in the hashmap %f and return those 13 bits (as a string of 0s and 1s) for all valid input strings of length 9 consisting of the three charachters , o and x where the number of xs is the same or one more than the number of os. I chose to include the empty input of nine s which returns thirteen 0s and this adds one more to the count of valid inputs: 6046 as I think the challenge wasn't clear about the all space input.

C (clang), 163 133 130 bytes

i,j,c,o,p,g;f(*a){for(i=c=1;c;){for(g=o=p=9,j=i++;p--;j/=3)o-=j%3,g&=a[p]=="x o"[j%3];c+=o*o==o;for(p=g*13;p--;c/=2)a[p]=48+c%2;}}

Try it online!

-33 bytes thanks to @emanresu A

Ungolfed:

void func(int* a) {
    int c = 0;
    for(int i=0; i<19682; i++) {
        int g = 1;
        int o = 9;
        int j = i;
        for (int p = 9; p--; j /= 3) {
            // Subtract the position (X or O) from o
            o -= j % 3;

            // Check if the current cell matches the input
            g &= a[p] == "x o"[j % 3];
        }

        // If the entire board matches (g == 1), generate the 13-bit binary of c
        for(int p = g * 13; p--; c /= 2) {
            a[p] = 48 + c % 2;
        }

        // increment c if o is 0 or 1
        c += o * o == o;
    }
}

K (ngn/k), 22 bytes

As suggested by emanresu A, a port of their Vyxal answer ends up a lot shorter:

1_2\"o x"[+3\<+/!9#3]?

Try it online!

K (ngn/k), 33 31 bytes

(13#2)\"x o"[+3\&~-2!+/1-!9#3]?

Try it online!

+3\&~-2!+/1-!9#3 generates all valid boards using 0 for x, 1 for spaces and 2 for o. "x o"[...] converts the boards to strings, ? finds the argument in the list of strings and (13#2)\ converts the integer to 13 binary digits.

Charcoal, 32 bytes

Ｐ×¹³0⮌⍘⌕ΦＥＸ³χ◧⍘ι xo⁹¬÷⁻№ιx№ιo²Ｓ²

Attempt This Online! Link is to verbose version of code. Explanation:

Ｐ×¹³0

Place 13 0s on the canvas so that the bits can be output LSB first thus avoiding having to manually pad them to length 13.

ＥＸ³χ◧⍘ι xo⁹

Generate all 3×3 boards of spaces, xs and os. (Some spurious "boards" are also generated but they will never match the input.)

Φ...¬÷⁻№ιx№ιo²

Filter on those where the count of xs minus the count of os is zero or one.

⮌⍘⌕...Ｓ²

Find the index of the input string and convert that to binary, LSB first.

JavaScript (ES6), 104 bytes

Expects a 9-character string made of ' ', 'O', 'X' and returns a 13-character binary string.

s=>(g=k=>i--?g(k/b|0)+"01O X"[t-=~-(k%=b),k+6%~b]:t="")((F=n=>g(n,i=9)!=s&&!t+!~t+F(-~n))(b=3),i=13,b=2)

Try it online!

JavaScript (ES6), 81 bytes

Expects a 9-character string made of ' ', 'O', 'X' and returns an integer in \$[0\dots6045]\$.

f=(s,n)=>(g=k=>i--?g(k/3|0)+"O X"[t-=~-(k%=3),k]:t="")(n,i=9)!=s&&!t+!~t+f(s,-~n)

Try it online!

JavaScript (Node.js), 100 bytes

s=>(i=1e4,g=(p,c)=>p[8]?i+=c&6?0:p<s:[0,7,1].map(n=>g(p+' xo'[n%5],~~c+n))|i)``.toString(2).slice(1)

Try it online!

Input a string, output 13 char 0-1 string.

JavaScript (Node.js), 77 bytes

s=>(i=0,g=(p,c)=>p[8]?i+=c&6?0:p<s:[0,7,1].map(n=>g(p+' xo'[n%5],~~c+n))|i)``

Try it online!

Input string as format 'x oxx o', output an integer in range \$\left[0,6105\right]\$.

We encoded all 6046 valid boards together with another 59 invalid ones into number not greater than 6105. The bit string output version also added an extra 1808 to each output so they actually encoded in range \$\left[1808, 7913\right]\$.

Ruby, 114 bytes

Brute-force solution. Iterates over all numbers in base 3, transliterates each to o/x/space, then if it’s a valid position increments a counter, breaking and returning the count in binary if it’s equal to the input.

->s{i=j=0
(t=("%9d"%j.to_s(3)).tr"012"," xo"
i+=1if(0..1)===t.count(?x)-t.count(?o)
j+=1)until s==t
"%013b"%(i-1)}

Attempt This Online!

Ruby, 104 bytes

Returns an integer rather than a binary string.

->s{i=j=0
(t=("%9d"%j.to_s(3)).tr"012"," xo"
i+=1if(0..1)===t.count(?x)-t.count(?o)
j+=1)until s==t
i-1}

Attempt This Online!

Haskell, 177 bytes

e s t=take t.g.d where d a=divMod a$l s;g(a,b)=s!!b:g(d a)
f=filter
l=length
o g f=(.g).f.g
a s=e"01"13.l.f(\u->o((o(\c->l.f(==c)$u)(-)'x''o')==)(||)1 0&&u<s)$e"ox "9<$>[0..3^9]

Try it online!

Brute force.

1. Generate all base 3 strings of length 9 using the alphabet "ox "

   e"ox "9<$>[0..3^9]

2. Count how many valid base 3 strings input is greater than

   l.f(\u->o((o(\c->l.f(==c)$u)(-)'x''o')==)(||)1 0&&u<s)$

3. Encode the result as a base 2 string using the alphabet "01"

   e"01"13.

Python 3, 121 bytes

lambda s:bin(g('').index(s)+8888)[3:];g=lambda p,c=0:[p]*(c%8<2)if p[8:]else sum([g(p+' xo'[n%5],c+n)for n in[0,7,1]],[])

Try it online!

Python port to my JS solution.

Python 3, 115 bytes

lambda s:bin([str(i).translate([32,120,111]*88)for i in rang(10**9)if{*str(i)}<={*"189"}!=i%9<2].index(s)*7)[-13:]

Try it online!

A very slow solution. The range is replaced by a customized implementation in tio link to speed it up. It loops every (decimal) number less than 10⁹, and only keeps numbers that is only composed with digits 1, 8, 9 and mod 9 less than 2. Then replace 1, 8, 9 by x, o, . And all these strings are considered valid and assign an encoding to it from 0 to 9348. Then multiply the encoding by 7 and mod by 2¹³.

Jelly, 16 bytes

3ṗ9SÞị“o x”iµBḊV

Try it online! Port of my Vyxal, see that for an explanation of why this works.

           i     # Find the index of the input in
 ṗ               # All combinations of
3                # 1...3
  9              # of length 9
    Þ            # Sorted by
   S             # their sums
     ị           # Index each value into
      “o x”      # "o x"
            µB   # Convert to binary
              ḊV # Remove first digit and concatenate

Python 3.8 (pre-release), 177 173 166 bytes

lambda s:f"{(t:=bin((L:=[f'{t:<9}'for i in range(3**9)if(t:=g(i)[:-1]).count('x')-t.count('o')in(0,1)]).index(s))[2:]):0>13}"
g=lambda x:'1'*(x<1)or' xo'[x%3]+g(x//3)

Try it online!

-11 bytes thanks to Malo

Basically brute force, converts base 3 to ' xo' strings and then enumerates cases where there are 0 or 1 more x's than o's. Also accounts for extra whitespace/zeros at the end of each string.

Ungolfed:

def f(s):
    L=[]
    g=lambda x:'1'*(x<1)or' xo'[x%3]+g(x//3)
    for i in range(3**9):
        t1=g(i)[:-1]
        if t1.count('x')-t1.count('o') in (0,1):L.append(t1+' '*(9-len(t1)))
    t1=bin(L.index(s))[2:]
    return '0'*(13-len(t1))+t1