【发布时间】:2016-07-05 19:27:59
【问题描述】:
我想将每个元组 X[p] 存储在不同的文件中 mini_batch1.jld 中的 X[1] mini_batch2 中的 X[2]..... 但是我下面的代码在创建的文件中存储(复制)了 X[p] 的所有元组。让我们看一个例子:
m= 100
k= 3 # number of tuples or partition
y=rand_gen(m,k)
(3,[[-1.0,1.0,1.0,1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,-1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0],[1.0,-1.0,-1.0,-1.0,1.0,1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,-1.0,-1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,1.0,-1.0,-1.0],[1.0,-1.0,-1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,-1.0,-1.0,-1.0,-1.0,1.0,1.0,-1.0,-1.0,1.0,1.0,1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,1.0,-1.0,1.0]])
我想拥有: mini_batch1 第一个元组
[-1.0,1.0,1.0,1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,-1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0]
mini_batch2 第二个元组
[1.0,-1.0,-1.0,-1.0,1.0,1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,1.0,-1.0,-1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0,1.0,-1.0,-1.0]
等等。但是,我的代码完成了创建 mini_batches 文件的工作,但未能一一存储元组。我该如何解决?
workspace()
using JLD, HDF5
function gen_random(m,k)
# m the length of the vector , for instance m=100000 and k
# the number of partitions let's set k=16
s = rand(m)
# Pkg.add("JLD"), Pkg.add("HDF5") these two packages are needed
# in order to store our vectors in files under the extension jld
# allow to convert each random number to -1 or 1
X=float_to_binary(s)
parts= kfoldperm(length(X),k)
# l want to store each tuple X[p] in a different file
# X[1] in mini_batch1.jld X[2] in mini_batch2.....
# but my code below store all the tuple X[p] in the files created.
for p in 1:length(parts)
file =jldopen(@sprintf("my path to file/mini_batch%d.jld", p),"w")
write(file, "X", [X[p] for p in parts])
close(file)
end
return [X[p] for p in parts]
function float_to_binary(s,level=0.4)
for i=1:length(s)
s[i] = s[i] > level ? 1.0 : -1.0
end
file = jldopen("/my path/mydata.jld", "w")
write(file, "s", s) # alternatively, say "@write file A"
close(file)
return s
end
function kfoldperm(l,k)
n,r = divrem(l,k)
b = collect(1:n:l+1)
for i in 1:length(b)
b[i] += i > r ? r : i-1
end
p = randperm(l)
return [p[r] for r in [b[i]:b[i+1]-1 for i=1:k]]
end
【问题讨论】:
-
write(file, "X", X[p])不会成功吗? (即,可能的问题是理解)。 -
不,它不起作用,因为它会读取第一个元组的第一个值。例如,如果 p = 4 。 X[1] 将是第一个元组的第一个值,X[2] 将是第一个元组的第二个值,X[3] 第一个元组的第三个值,X[4] 第一个元组的第四个值元组。但我正在寻找这个结构 X[1] 整个第一个元组,X[2] 整个第二个元组等等
-
y=gen_random(m,k) # m = 100 , k =4 例如 (4,[[1.0,1.0,1.0,1.0,1.0,-1.0,-1.0,-1.0, -1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0],[1.0,-1.0 ,1.0,-1.0,1.0,1.0,1.0,1.0,1.0,1.0,-1.0,1.0,1.0,-1.0,1.0,-1.0,1.0,-1.0,-1.0,1.0,1.0,-1.0,-1.0 ,1.0,1.0],[1.0,-1.0,1.0,1.0,-1.0,1.0,-1.0,-1.0,-1.0,1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,1.0,- 1.0,1.0,-1.0,-1.0,-1.0,-1.0,-1.0,1.0],[1.0,1.0,-1.0,-1.0,-1.0,1.0,1.0,1.0,-1.0,-1.0,1.0, 1.0,1.0,-1.0,-1.0,-1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,1.0,-1.0,1.0]])
-
这是我想在每个文件中获取的元组的结构:文件 1 => 元组 1 [1.0,1.0,1.0,1.0,1.0,-1.0,-1.0,-1.0,- 1.0,1.0,1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.0,-1.0,-1.0,-1.0,1.0,-1.0,1.0,1.0,-1.0] 等等。希望它可以帮助您更好地捕捉问题!